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Preface 



This is the third of a series of monograpi > be published by the Rehabili- 
tation Research and Training Center in Mental Retardation at the t^iversity 
of Oregon. One of several such Centers which are supported, in part, by grants 
from the Social and Rehabilitation Service of the Department of Health, h.du- 
cation and Welfare, the Or-gon Center conducts research related to the re- 
habilitation of the retarded and provides training for professional personnel 



engaged in the rehabilitation process. 

This monograph grew out of a training seminar held in February, 1969, at 
the University of Oregon entitled, “Strategies for the Evaluation of Short- 
Term Training.” The program was designed for rehabilitation personnel 
engaged in administering short-term training programs. The major objective 
of the conference was to introduce the trainees to various strategies o nro-- 
gram evaluation. The editor wishes to express his appreciation to ^ x., ' 

Chyatte Dr. Windle Dickerson, Dr. Robert LesUe, Dr. Patrick Flamgan, Dr. 
Bryan Smith, and Mr. Robert Mil!;r, the conference speakers, for their assist- 
ance in making the program a successful one. i i 

It would be impossible to express appreciation through mdividual acknowl- 
edgment of all those involved in one way or another in the preparation of t^s 
monograph. However, a special thanks goes to Miss Cheryl Mulder who 
supervised the production of the manuscript. Finally, the editor wishes to 
express his appreciation to the authors and publishers who gave permission for 
the reprinting of the materials. 



P.L.B. 



Foreword 

The appearance of a monograph on this subject is a welcomed resource in 
education for rehabilitation. It focuses on a major concern that has long 
plagued both trainers and administrators. ... , 

Past efforts to investigate the effectiveness of short-tenn training have been 
more creditable than credible, for myriad research problems confront the 
investigator— and almost as many critics. A monograph on the evaluation or 
training in rehabilitation, therefore, would have been welcomed at any time. 
Its appearance now is particularly timely, as new approaches to the manpower 
gap in reht bilita*. ion services turn increasingly to inservv''’ trainm- 
study programs, and various f<^- rt-term trn .kint. 

Difficult as it is, lun is . aiy for the conduct ot any effective 

training. Xhe evaluation of training that relates directly to a joD being regularly 
performed is particularlv difficult. For instance, even when an obseiwable 
improvement in job performance could be noted, it would remain to be shown 
thi^ it was a particular period of training that had brought ^out the change. 

Adminiscrators -with their necessary concern for the cort-benefit aspects of 
thr*r programs must inevitably seek evidence that a training program was 
wcirrh either the cash outlay or loss of time of their staff members. Xh^ often 
turn to th.i trainees themselves with requests for evaluation: What did you 
get out of it? Should I send others, too? Or they turn to the trainers. Could 
you do the same thing in less time? Could you set up a course clos« by so we 
woiKidn’t have to travel so far? Or could you do something different— or 

Xradners, on the other hand, are justifiably concerned the “I- j 

porticxilar course or a particular approach on the trainees. Xhey ask what ^d 
you get out of the course? What changes would you make? When asked for 
their own evaluation, they often respond with the answers they have received 

^ who ^e in the training business know all too well that neither these 
answers nor these questions are adequate for evaluating a traimng program 
or ei-en a single course. We know they provide valuable tramce reactions 
which can help trainers improve the teaching or increase the learning poten- 
tiaL We are well aware that even the most conscientious trainer rarely has time 

or inclinanion to go further than this. . , j t. i j 

O One would hop<*. that the material on evaluation described here would en- 



courage trainers to attempt more precise studies of their training program; 
for SKnce, that they would develop ways of dcscnbing the desired foaming 
outcomes in less global terms than one usually sees m applications for short- 
term training funds. What a relief it would be to the grantor of funds for 
training to have goals set forth in measurable or observable terms, reasonable 
ones that could be achieved in a two- or three-day course. 

An equally difficult research problem we would hope to see encour^ed is 
that of individual response to training. Here one is keenly aware of <:he ad- 
ministrator’s stake in his staff development. We would not look 
a time in the near future, if ever, when an administrator could have a 
tell him which counselor to send to which course at what time, btili, strategies 
of evaluation which take seriously individual responsiveness couldbe of great 
help for long-range planning for short-term training. 

Finally then, in looking forward to the appearance of this monograph, we 
would ho’pe that it signals another step toward that long-range planning with- 
out which any evaluation of short-term training remains inconclusive. In our 
mind it is not the evaluation of a single course or even a specific training and 
learning that will eventually provide rehabilitation services of the quality we 

all seek. Margaret F. Clark 

Consultant, Short-Xerr>^ 
Training 

Division of Training; 

RSA, SRS, HEW 
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Introduction 

Rehabilitation is a concept: wliich has assumed an increasingly influential 
role within the helping professions over the past several decades. In alignment 
with this trend has been the establishment of short-term trainmg programs 
designed to meet certain manpower needs. The ultimate goal of these pro- 
<rrams is to provide more effective rehabilitation services to the handicapped. 
In order to help maximize the impact of this rapidly expanding area of short- 
term training activities in rehabilitatic ^: considerably more attention needs 
to be given to the following general proposition: ^ 

The extent to which effective rehabilitation services are delivered to the 
handicapped is, in parr, a function of the extent, to which short-term . 
training is effeedve. 

If the above proposition has any validity, then it becomes the responsibility of 
professional training personnel to make every effort possible to improve the 

effectiveness of such programs. . j z: • i i. u 

This monograph reflects the position that one such effort defiimely should 
be through the EVALUATION of training. To set the stage for the materials 
that follow, the terms rehabilitation, short-term trainmg and evaluation m:e 
discussed. In addition each of the sources contamed .within the monograph 
is brieflly introduced. 



RehafcMitatlon 

Obermann ( 1965) defined the term “rehabilitation” 3S “that activity that is 
required to assist an [handicapped] individual to move from a smtus of inade- 
quacy to a status of adequacy” (p. 41). At one time “that activity or service 
was provided by a limited number of discipUnes which operated m a somewhat 
segregated manner. The current r .end, however, is to mtegrate the seivices o- 
the medical, psychological, sociological, educaoonal, and vocatioiwl 
sions to provide maximum rehabilitation (Lofquist, N. D.)._ Although rel^b - 
ration services were previously limited primarily to handicapped individuals 
with phy sically disabling conditions, they have now be^ expanded to “delude 
mental, social, economic, and educational disabihoes. The rehabili- 

tation have also undergone dramatic change* ^Vhen the field was irainly 
Q ocationally oriented, the disabled person’s inadequacy-adequacy status simply 

ERIC . « 
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referred to unemployed-employed. The current ° 

quacy however, is the “total” adaptation of the individual ptrauss, 19.. ). 

In order to truly capture one’s imagination regarding the growth and 

d^e^otment; ^n"d "t^^i^d^ o^^ 

vided for rehabilitation efforts for the years of 1920 and 1968, re^^tiyeiy. 
As for the number of individuals served, only 532 

the first year of the federal-state program whereas 208,000 people were sue 

cessfiillv rehabilitated in I9d8. « -i- — 

In suite of the tremendous gains made in the field of rehabilitation there 
are certain problems which continue to prevent maximum rehabilitation 
services One of the most i.r^ent and critical of these problems is ^ 

manpower. As recently as 1968 a National Advisory Committee issued the 

following statement: 

We believe that appropriations for support of training should be ^bstan- 
tially increased if they are to be even nearly commensurate with the mag- 
nitude of manpower needs and the resources of educational institutions 
and cooperating agencies for the preparation of personnel the profes- 
sional fields most closely related to rehabiliwtion of disabl^ 

(Report of the National Citizens’ Advisory Committee on Vocational 

Rehabilitation, 1968, p. 56.) 

The first major attempt to confront the manpower problem was in 1954 
through Public Law 83-565 which provided for grants to support the rtainmg 
of more professional personnel for rehabilitarji:.-n services. Of five typ 
grants provided, one was for short-term training. 
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Short-Term Training 

This type of training can be viewed in several ways; for our purpose, how- 
ever^ the following definition is offered: 

Training that is of shorter dura don than a quarter or a ^inester of a 
regular academic year is generally regarded as short-term traimng. It is 
inrtnded to precede or supplement basic or advanced professional educ 
t?on on an academic or calendar year basis. Short-term trainmg courses 
generally range in length from 2 or 3 days to 6 weeks, according “ ^heir 
Specific VurpSse. They include institutes, workshops, seminars and other 
training courses (Support of Short-Term Training, , P- /• 

When the training program commenced in 1954, $900,000 Avas appropriated 
for training grantf In 1967 a total of $29,700,000 was appro pria^d (Report of 
the National Citizens* Advisory Committee on Vocational Rehabilitation, 
1968) wSS a ^bstantial portion of this designated for short-term traimng 

^'^A major objective of the short-term training program has been to Provide 
rehabilitation personnel already m the field with a 

methods and philosophy of rehabilitation, as well as to afford thum an oppor- 
tunity to raise their level of knowledge and skill. In essence, short-term train- 
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ing is one means of attempting to facilitate the effectiveness of rehabilitation 
services provided by personnel serving the handicapped. 

The sconr of th.c program is partly reflected by the disciplines it serves. For 
example, short-term training grants are now made available to the fields of 
physical medicine and rehabilitation, physical therapy, occupational therapy, 
speech pathologv and audiology, rehabilitation nursing, rehabilitation social 
work, prosthetics and orthotics, rehabilitation psychology, rehabilitation 
counseling, recreation for the ill and handicapped, as well as other specialized 

Helds. . . , 

In 1959 short-term courses and institutes reached only 2,500 professional 
personnel already in the field; in 1968 the Rehabilitation Services Administra- 
tion funded this type or training so that more than 9,000 individuals were served 
(Hunt, 1969). In addition, the Regional Rehabilitation Research and Training 
Centers in 1967 conducted 364 short-term training courses which sci-ved 
18,415 trainees (Report of the National Citizens’ Advisory Committee on 
Vocational Rehabilitation, 1968). These figures do not include the short-term 
training courses offered through the states’ rehabilitation in-service trainmg 

programs. . 

Of course, the extent to which the short-term training programs adequately 
meet certain manpower needs depends on the effectiveness of the individual 
programs. Thus, it becomes the responsibility of each traimng director or 
persons in charge of short-term courses to make every effort to imprtwc their 
program so that the trainees will be able to provide more effective rehabilita- 
tion services to the handicapped. As already indicated, one such effort defi- 
nitely should be through the evaluation of training. 



Evaluation 

Although attempts to treat this term have been many and varied, a major 
failing in evaluation has been said to be a lack of adequate definition (Alkin, 
1969). The following definition is presented primarily because it recognizes 
evaluation as a process that involves the identification of many factors that 
contribute to educational or training outputs.*^ 

Evaluation is the process of first identifying and then quantifying or meas- 
uring the relationships between student [trainee] inputs and educational 
[training] outputs and determining the combination of mediating factors 
which maximizes the educational [traimng] outputs, given a consrant 
financial input and controlling for the effects of external systems (Alkin, 
1968, p. 1). 

Trainee inputs refer to the nature and characteristics of those entering a 
training program, e.g., type and level of professional fining, motiyaoom 
Alkin speaks of educational or training outputs as referring to (a) botfi 
cognitive and non-cognitivc changes that the person experiences following an 
instructional program, and (b) program impact upon systems external to it, 
c.g., agenc>’, community, ,<r _ 

Mediating factors arc descriptive characteristics of the program itself, c.g., 
personnel, curriculum, small gfroups, audio-visual aids, practicum cxj^rtcnce 
Financial inputs simply refer to the monetary resources made available for 
carrying on the program. Fin ally, external systems mean the framework of 

1 The reader is referred to Moss* and Alkin’s articles in this booklet for t«vo additional 
O ‘finitions of evaluation. 
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social, political, legal, economic, and other systems outside the training pro- 
gram which encompass the program, have impact upon it, and are in turn 
modified by the outputs of the program. Xhe reader is referred to Alkin s 
(1968) paper for further discussion and a figurative presentation of this defi- 
nition. Since there is a diversity of opinion regarding what coi^tutes evalua- 
tion, this definition represents only one of many and varied viewpoints. 
General agreement probably exists, however, as to the important purpose it 
serves in a training program. 

Tracey (1968) stated that evaluation served to determine: 

1. . . . where the activity is at any given moment and providing a baseline 
for measuring progress. 

2. . . . the value of training and development program activities to the 
enterprise and of appraising the efficiency and effectiveness of the 
functions performance of the task set forth. 

3. . - . whether the time, ener^, and money expended in planning and 
operating programs of training and development are producing results 
sufficient to justify the investment (pp. 12-13). 

Tracey also listed three functional ways in which an adequate evaluation 
program is critically important to training: 

First, the steady growth of training and development activities in most 
enterprises, which in total involve millions of people and many more 
millions of dollars, makes it essential that those responsible for the manage- 
ment of these activities be able to defend their programs by knowing the 
accomplishments and contributions of the activities to the enterprise goals. 
The continued support of top management hangs in the balance. Second, 
evaluation provides trainers with a means of determming the efficiency, 
effectiveness, and utility of both management and operation. Only by 
appraisal is it possible to ensure that programs are suited to the groups for 
which they are designed and that they result in the behavioral changes 
required for improved products or services. Third, evaluation provides a 
starting point for the design of an improvement program (p. 13). 

Finally, he set forth several fundamental assumptions which underlie the need 
for evaluating training: 

!. Any training and development program must be validated; that is, the 
efficiency and effectiveness of programs must be objectively deter- 
mined. They must be subjected to critical evaluation and must demon- 
strate their value to the organization if they are to be retained. 

2. Any training or development program can be improved— no program 
is perfect. Although the effectiveness of the program may have been 
den<onstrated, further refinements arc possible. 

3. Improvement of any training or development program can be affected 
by: 

a. objective and coordinated evaluation of every aspect of the oper- 
ation. 

b. the application of imagination and creative thinking by all personnel. 

c. deliberate collection of the observations, ideas, and thinking of all 
personnel. 
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d. critical analyses and synthesis of findings, ideas, and alternatives. 

e. systematic, time-phased development and tryout (Tracey, 1968, pp. 
13-14). 

If short-term training is going to have a significant influence, then greater 
attention certainly must be given to the evaluation of such training activities. 



OVERVIEW OF THE MONOGRAPH 

This monograph is not intended to serve as a comprehensive overview of 
evaluation. Consequently, the reader is pr^ented with only a sample of 
evaluative theoretical positions, methodological approaches, and innovative 
strategies. Nevertheless, it is hoped that what is contained within these covers 
will serve as an “eye opener” and an incentive to more and better evaluation 
of short-term training in rehabilitation. 

Marvin C. Alkin, Evaluation 'Theory Development 

Alkin presents only one of a number of evaluation theories or models which 
have been offered. In addition to being introduced to an evaluation model 
which is couched in a decision-making framework, the reader is presented a 
definition of evaluation and five evaluation need areas. _ i • i.- 

Evaluation is a complex process which can represent the inteCTelationsmp 
of an infinite number of parameters or variables. Thus, evaluation theories, 
systems, or models are needed to enable us to reduce the complexities of 
evaluation to more manageable proportions. 

Jerome Moss, Jr., The Evaluation of Occupational Education Frograms 

Moss presents an evaluation model which is counter to the often rai^d but 
somewhat misleading quc.stion, “Is short-term training effective?” This type 
of question presents a number of difficulties. In the first place, it fails to 
consider the input of differential trainee variables. Secondly, it often views 
training (the independent variable) as a uni-dimensional entity. A more 
meaningful question that training personnel should be asking, and one upon 
which Moss’ model reflects, is “What kind of trainees (input), given what 
kinds of training experiences (process), demonsttate what kinds of change 
(output)?” Ill addition to the differential evaluative model. Moss discusses a 
number of important considerations in evaluation, e.g., definition, outcome 
criteria, types of research approaches. 

Donald L. Kirkpatrick, Evaluation of Training 

The author discusses an approach to evaluation which involves what he 
proposes to be the four major steps in the evaluative process, i.e., reaction, 
behavior, learning, and results. In addition, the reader is provided with guide- 
lines, procedures and examples of the four steps. Even though the examples 
are taken from the fields of business and industry, it is important to stre» that 
the guidelines, procedures and techniques described are certainly apphcable 
to the evaluation of short-term training in rehabilitation. 

Bryan Smith, On Going Program Evaluation 
0 Evaluation is often viewed as being limited to a pre-post test approach for 
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the purpose of assessing the effects of training. However, most experts m 
the field now take the position that evaluation is an on-gomg process and that 
the program planning phase is one of the more important evaluative stages 
to be considered. Beckard (1962) stated, “all planners can improve the effec- 
tiveness of their meetings by building into their planning ^stemaac ways 
of appraising the situation as they go along (p. 52).” Smith presents the 
reader with a systematic tool for program planning and oii-going evalua- 
tion — Program Evaluation and Review IT echnkiue (PERT) - 



Leo A. Hamerlynck, Behavioral Criteria for Short-T erm Training 

The type of outcome criteria we choose to select for evaluating the 
tiveness of short-term training in rehabilitation is of critical importance. This 
author presents a categorization model which includes four types of criteria 
and discusses the implications of each. The direct-prima^ type (actual client 
behavior) of outcome criteria has been ofen ignored yet is most germane since 
the ultimate purpose of rehabilitation short-term training is to better help the 
handicapped. 

Patrick J. Flanigan, Professional Opinions Regarding Curriculum Content in 

Short-Term Training Programs in Mental Retardation: An Evaluauon 

Survey 

The author presents a descriptive summary of a curriculurn ev;^ation for 
rehabilitation short-term training programs in mental retardation. The survey 
was based, in part, on the :issumption that enrichment and modification of 
short-term training programs are most realistically brought about by consulta- 
tion with professional people in the field. Over a four year span, 3,299 Prac- 
titioners representing 12 professional orientations were screened regardmg 
their suggestions for specific curricula which they thought was applicable for 
short-term training programs in mental retardation. 



Evaluation of Training: Annotated Bibliography 

Ninety-five annotated references are provided on the eyalu..tion of training, 
most of which have been derived from the fields of business and education. 
The reason for this is that the rehabilitation literature is practically nil on the 
subject with the exception of the brief attention given to the area by the 
Institute on Rehabilitation Services (IRS). 

Personnel in the fields of business and industry have been largely respon- 
sible for the advancements made in training technology, and have alw been 
leaders in the area of evaluation of training. As well, evaluation special^ arc 
now emerging in the field of education which is indicated partially by the 
UCLA Center for the Study of Evaluation and the Ohio State Umversity 
Evaluation Center. The time has come when rehabilitation training personnel 
must begin borrowing training assessment strategics from these and other 
professional fields. 
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Evaluation Theory Development’^ 

Marvin C. Alkin, LTniversity of California, Los Angeles 



The development of a theory of evaluation has been established as the major 
goal of the Center for the Study of Evaluation. A theory of evaluation should: 
(1) offer a conceptual scheme by which evaluation areas or problems are 
classified, (2) define the strategics including kinds of data, and means of analy- 
sis and reporting appropriate to each of the areas of ths conceptual scheme, 
and (3) provide systems of generalizations about the use of various evaluation 
procedures and techniques and their appropriateness to evaluation areas or 

problems. t. u 

At their best the propositions presented in a theory of evaluation should 
enable one to predict, fully, the appropriateness of utilizing various evaluation 
strategies within a system. Development of an evaluation theory is thus an 
“end” rather than a means, guiding the research activities of the Center for 
the Study of Evaluation. Development of a theory is a difficult enterprise. 
The process of working toward the achievement of this end requires a con- 
ceptual framework to guide and coordinate our efforts. What is presented in 
this paper represents months of conceptual efforts and may be thought of as 
a first approximation of an attempt to develop an evaluation theory. In other 
words, we have constructed a rationale for conducting evaluations in a cermin 
way that is based on a specified set of assumptions which in turn underlie a 
precise definition of what an evaluation is supposed to do or be- 

To start with, in the development of a theory, it is necessary to reach agree- 
ment on a definition of evaluation. JVlost would agree that a major failing of 
evaluation today stems from the lack of an adequate definition. Past defini- 
tions of evaluation have either equated it with; (1) measurement and testing, 
(2) statements of congruence between performance and objectives, or (3) pro- 
fessional judgments. None of these definitions by itself is sufficient to provide 
jiH the necessary information or to include the multiplicity of activities now 

regarded as evaluation. ^ . 

In the past year, there has been in .easing evidence of a developing con- 
sensus on a broader, more comprehend vc definition of evaluation. This ex- 

1 Published in Evaluaticn Comment, October 1969, 2, (1) 2-7, and reprinted with the 
n^rmJssion of thc autHor« 
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panded v iew has necessarily taken into consideration that the judgments from 
evaluat ors are intended to be of use to decision-makers in selecting among 
various courses of action. This view of evaluation also acknowledges the 
uniqueness of specific situations or programs and the necessity of recognizing 
this uniqueness in the evaluation as well as in the manner in which the evalu- 
ation information is ultimately reported. • 



1. Evaluation is a process of gathering information. Most past definitions 
of evaluation are inadequate since they do not cover the full range or 
activities requiring information. 

2. The information collection in an evaluation will be used mainly to make 
decisions about alternative courses of action, rather than being em- 
ployed in some other fashion. Thus, the manner in which the informa- 
tion ' collected, as well as the analysis procedures, must be appropriate 
to the needs of the decision-maker or of potential decision-involved 
publics. This requirement might necessitate quite different analyses 
than those which might be employed if the purpose were understanding 
the education process per se. 

3. Evaluation information should be presented to the decision-maker in 
a form that he can use effectively and which is designed to help rather 
than confuse or mislead him. 

4. Different kinds of decisions may require different kinds of evaluation 
procedures. 

While there are any number of variations of a specific wording that 
serve equally well for a definition of evaluation, we have devised one which 
fits our conceptions of evaluation and meets our biases. We would maintam 
that evaluation must take into consideration the ultimate decision-making 
functions to be served, as well as the nature of the specific problem or situation 
under analysis. We prefer the following definition; 

Evaluation is the process of ascertaining the decision areas of concerUy 
selecting appropriate information, and collecting and analyzing infomm- 
tion in order to report summary data useful to decision-makers in selecting 
among alternatives. 

The first part of the definition of evaluation presented here deals with 
ascertaining the decision areas of concern. The decision-make^ and not the 
evaluator, determines the nature of the dom^n to be examined T^ evaluator 
can and should, however, point out inconsistencies, potential dimculties, or 
additional data that might modify the decision-maker’s views on the relevance 



of certain concerns. 

For example, if the evaluator is called upon by a specific decision-maker to 
provide an evaluation, first, he will want to know what should be evaluated. 
Decision areas of concern may be stated relative to explicit statements of goals 
or objectives of the system or relative to various imphcit goals. In his mter- 
actions with the decision-maker, the evaluator may wkh to point out tl^ 
necessity for broadening the area of concern because of mterrelated aspects 
of the school [training] program, or to consider, as well, various areas of 



potential unanticipated outcomes. , , • r 

Or. the other hand, if the evaluator is conductmg an evaluative study of 
an educational institution without having been commissioned by a specifac 
decision-maker, he has available greater flexibUity. A professor, for example. 
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might conduct an evaluative study of his university. There is a p econception 
on the part of the evaluator as to which decision-maker or potential decision 
groups he is directing his work towards. Thus, the decision area of concern 
in such an endeavor is framed by the unique nature of the potential decision- 
maker or decision groups along with either actual data or judgments on the 
part of the evaluator as to the concerns of this group (individual). 

We consider this “preconception of decision-maker’’ notion a fundamental 
and useful distinction between evaluation and some kinds of research- If one 
realizes that the purpose of what he does is to provide the best possible basis 
for informed judgments or decisions, his thinking about his task will surely 
be influenced; and this will be a different influence than that which operates 
on the researcher whose purpose is to discover or explain some phenomenon. 

Another part of the definition and, therefore, another task of evaluation 
deals with selecting appropriate information in light of the decisions areas to 
be considered- If the decision area relates to the assessment of the needs of a 
total system, the information requirements will be quite different than when 
the decision area is related to the relative success of two specific alternative 
programs conducted under experimental conditions- The task of the evaluator 
in specifying information requirements includes the development of the evalu- 
ation design of the project, and the selection and/or development of instru- 
ments designed to provide the information appropriate to the decision areas- 
Collecting and analy%mg the information are tasks of prime concern to the 
evaluator- He will encounter different problems associated with these tasks, 
depending upon the unit being evaluated, the nature of the decision-maker, 
and other considerations- 

One of the most vital parts of the evaluation process is reporting summary 
data to the decision-maker- Most evaluators often overlook this function as 
being merely a pro forma exercise- The evaluator’s role requires that he make 
judgments about the relative worth of various courses of^action- These judg- 
ments may be in the form of statements or recommendations to the decision- 
maker (s), or may be general descriptive material. But in all instances the 
evaluator should attempt to be explicit in the specification of the value system 
that led to the judgments m ide- Indeed, if the purpose of evaluation ^ to 
provide information that w^ill enable decision-makers to reach decisions about 
alternatives, then the nature and form of the reporting should be appropriate 
to the problem and the audience. 

The summary data is provided to be of use to the decision-Tttaker. It has 
already been aUuded to in this paper that we are using the term “decision- 
maker” to apply both to an explicit contractor of evaluation sendees as weU 
as a potential but only implicit deciiion-maker or group. Moreover, we are 
using the term “decision-maker” to apply both to an individual with organiza- 
tional “line” authority (e.g., a school principal [a training^ director] ) as well 
as to other publics that participate in the decision process or in the development 
of educational policy decisions. Tluroughout this paper, whenever we refer to 
“decision-maker” it is in the generic sense discussed above. 

Information is provided to “decision-makers,” in order to enable sounder 
decisions in selecting among alternatives. By definition, a “decision’’ involves 
making a choice among alternatives. However, the form of alternatives has a 
wide range. Alternatives may range from a “go/no-go” category regarding 
a given textbook for a particular classroom to a complex aggregation of a 
number of budget categories related to an optimum expenditure level. In 
general, the number of categories of alternatives increases as the size of the 
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program or system increases; e.g., pupil achievement, teacher morale, teacher 

summary evaluation data should ordinarily be presented fo^ 

of statements and/or recommendations about alternatives. An exception 
be when such information is designed to describe the status (past, present or 
?mure) of the system. For example, “the students at your 

mathematics.” In this instance, there are no alternatives and the decisions a 
implied (e.g., something should be done to correct this situation). 

EVALUATION NEED AREAS 

The foregoing definition and assumptions are closely 
making process, which in turn leads to a consideration of what l^^^s of edu- 
cational decisions require evaluative information. 

has led to the development of a decision-oriented classification of the various 
types of evaluation. Five areas of evaluation may be 
^ These five areas represent attempts to provide evaluative infomation 
satisfy unique decision categories. In other words, there are evaluations 
sary m providing information for decisions about the state of the system. 
ca^such evaluations systems assessrnent.') There are evaluations necessa^ m 
providing information to assist in the selection of particular programs likely 
to be effective in meeting specific educational needs. (We call this Icind of 
evaluation, which takes place prior to the 

program planning.^ There are evaluations nec^sary in providing 
relative to the extent to which a program has been introduced in 
in which it was intended and to the group for which it was intended 
imple?nentationy. There are evaluations necessary in Providing info^auon 
during the course of a program about the manner m which the program k 
fimclfoning, enroute objectives are being achieved, and what unanticipated 
outcomes fre being produced. Such information can be of value in «i<>difying 
the program {prograr/i improvc7ne7it') . Evaluations are necessary m providing 
information th^t might be used by decision-makers in making judgments about 
the worth of the program and its potential generalizability to other related 

“*^The°^a1SaSSI«e^SS^^^ seem to represent a growing 

among a number of people engaged in the study of evaluation. The first two 

and the last need areas discussed are somewhat sirmlar, 

text.” “input,” and “product” presented by StufF.ebeam (1968). What he 
refers to as “process” we have chosen to think of as two separate stages, pro- 
gram implementation and improvement. Major 

found bet-w-een the descriptions of our need areas and his ®^Ses. Howeve^ 
Stufflebeam’s work contributed substantially to our thmkin^ The discre^n y 
model presented by Provus ( 1969) outlines five stages:^ definition, t-h-el 

tion ” ‘process,” “product,” and “cost benefit analysis. The f^st four of these 
«Tsom?what similar to our stages two tl^ough five. Provus does "pt 
systems assessment as a part of his model. With respect to cost benefit analys^ 
we would maintain that cost benefit cmnsideratoons are a part 
area of the evaluation and cannot be thought of as simply an additional lask 
to be attacked when all of the other evaluations have been completed The 
general notion that cost benefit consideration arc 

evaluation process also would seem to be subscribed to by John Hemphm m 
his recfint NSSE Yearbopk chapter (1969). The evaluation types outlmed by 
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Rodney Skager (1969) are also quite similar to the framework presented here. 
A. better understanding of the need areas will be gained by a more complete 
description of each. 

Systems Assessment 

Systems Assessment is a means of determining the range and specificity of 
educational objectives appropriate for a particular situation. The ^eds ^ay 
be represented as a gap between the goal and the present state of affairs. The 
evaluative problem then becomes one of assessing the needs of students, of the 
community, and of society in relation to the existing situation. Assessment, 
therefore, is a statement of the status of the system as it presently exists m 
comparison to desired outputs or stated needs of the system. 

A systems assessment might be related to evaluation of a specific instruc- 
tional program and thus the charge would be to detemine the present status 
relative only to a specific objective and related objectives. We would refer to 

this as a “sub-system assessment.” _ _ . , 

Systems assessment does not refer to specification of process charactei istics 
appropriate for a district, school, or classroom. A statement such as “this dis- 
trict needs a lower pupil-teacher ratio” or “a need of this district is to install 
team teaching” is not a systems assessment. A 'he systems assessment must be 
related to the ultimate behavior of clients of one type or^ another (pupils, 
parents, community, etc. — all clients of the school). To put it simply, systems 
assessment must result in a statement of objectives In terms of outputs of the 
school [training program]. 

The process in the systems assessment area of ascertaimng the decision area, 
specifying and collecting information and reporting summary data, requires 
methodology and techniques different from that which might be employed, 
for example, in a typical experimental design. The data are concerned with 
the status of the ^stem. The summary data might be comparative, historical, 
or other descriptive information. 

Program Planning 

Program planning, the second need area, is concerned with providing infor- 
mation which will enable the decision-maker to make planning decisions--to 
select among alternative processes in order to make a judgment as to which 
of them should be introduced into the system to fill most efiiciently the critical 
needs previously determined. In an instance where we are proceeding through 
severe need areas in sequential fashion, the follow ving might occim. After the 
decision-maker receives the systems assessment evaluation, he might make a 
decision as to the appropriate means of fulfilling that need. Alternatively, he 
might designate several possibilities and ask the evaluator to provide informa- 
tion on the possible impact of each. Hence, in program planning, tlie evaluator 
provides the data for an evaluation of a program prior to its inception. The 
task of the evaluator is to anticipate the attainment of goals and to assess the 
potential relative effectiveness of different courses of action. 

It is quite obvious that the collection and analysis of data of the type re- 
quired for this evaluation need area will be quite different from collection 
and analysis problems for other areas. The techniques may require both mternal 
and external evaluation procedures. (See Lumsdaine, 1965.) 

By way of internal evaluation, programs may be examined to detei-mme the 
extent to which their reproducible segments purport to. achieve the objectives 
of the program being evaluated. Technical feature^ ^ style or construction, 

O 
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oracticality and cost are other means of providing internal evaluation. To 
dSe, the Evaluations of products by EPIE have been primarily based upon 

of programs yet to be implemented might take the 
fomi of examining research data on the results of implementation in similar 
or near-similar situations. Or external evaluations might attempt to utilize 
some of the various educational planning techniques to obtain data. Computer 
simulations might be developed; Delphi analysis might provide insights into 
the potential outcomes of a program; gaming and v^ious other systems 
analytic approaches might also provide external evaluation data. 

Program Implementation 

After the decision-maker has selected the program to be implemented an 
evaluation of program implementation determines the extent to which the 
Siolememed program meets the description formulated m the prograrn plan- 
SS dlc7“on In the case of an exiting program where m™S.e"5 r 

have been implemented, the evaluation task ax this stage is ® , with the 

dee-ree to which planning descriptions of the program coincide with the 
taSememed progmm and the extent to which assomed descriptions of mpnta 
to the svstem {students) correspond with observed inputs. 

TheE haEe been numerous examples in the educational literature of con- 
flicting results relative to the impact of a specific instructional treatm^^ 
would maintain that in large part this is attributable to the lack o sp ^ ^ 
of the precise nature of the instructional treatment that was 1 

teaching is not always team teaching. More precisely, t^m teac i g 
Rosa ndght be quite different from team teaching m Boston or from team 
teaching^in Palo Alto, California. The precise defimtion or the paramete. 
defin^d^as team teaching in a giveri situation would help to insure an under- 
handing of what is being evaluated and whether what is being evaluated is 
what the investigator thought the program was. 

Program Improvement 

The evaluator can play an important role in program 
fo"ich need area, by providing as much information as possible about the 
success of the parts of the program. In order to perform program im- 
provement tl recognize the bxsiclly intervent.on«t 

rc'ilf* that the evaluator has been asked to take. . ^ 

The key point in the understanding of ths role of the evaluator in perfomi- 
in JhlaluTtiSns in Siis need area is that he is first and foremost an inte^enuon- 
LsEattempting to provide data which will lead to the immediate modificatmn 
and ho^fully, improvement of the program. As the evaluator identifies 
nrobleSS and collects and analyzes related information, data are presented 
SJimSiely to the decision-maker so that changes may be executed within 
the system^to improve the operation of the program. InformaUon inight 
incline data on the extent to which the program appears to be achiev- 
ing the prescribed objectives, as measured by regular tests; information 
Sfght be presented which relates to the impact of the program on other 

'"''ThSEeEd^I^fah^J often been overlooked or ignored*by the teaditional 
evaluator who has attempted to impose the antisepuc rterihty of the labora- 
teS^^on the real world. Such aft approach may make for a fine experirnem, 
biiiT it does little to improve a program which is often not m its final form. 
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Program Certification 

In the fifth evaluation need area, program certification, the role of the 
evaluator is to provide the decision-maker with information that will enable 
him to make decisions about the program as a whole and its potential general- 
izability to other situations. The evaluator might attempt to provide informa- 
tion which will enable the decision-maker to determine whether the program 
should be eliminated, modified, retained, or introduced more widely. 

The kind of information collected for program certification decisions is m 
large part dependent upon who is the intended decision-maker. It is obvious 
that different information will be required if the potential decision-maker is 
the teacher, the principal, or a funding agency. Evaluations in this area will 
be concerned with examining the extent to which the objectives have been 
achieved, as well as with the impact on the outcomes of other programs. 

In program certification evaluations, there is a requirement for valid and 
reliable data which would generally require that the evaluator attempt to apply 
as rigid a set of controls as possible. The evaluator might use pre- and post- 
test designs and employ sophisticated methods for analyzing the data. Inter- 
vention should be avoided in evaluations in this need area. Here the traditional 

evaluator is “at home.” _ , . . 

In considering the situations in "which evaluation might take place in various 
need areas, we have found it helpful to differentiate between the evaluation 
of educational systems and the evaluation of instructional programs. In terms 
of the conceptual framework that has been presented, one can view the evalu- 
ation of educational systems as involving the first two need areas and the 
evaluation of instructional programs as largely involving the last three. 

In evaluating any educational system it is necessary to determine the edu- 
cational needs in terms of the most appropriate objectives for the given system 
and to devise a procedure for providing regular information on the progress 
of the system relative to these dimensions. "This procedure is the evaluative 
device for decision-making about the assessment of system needs (Systems 
Assessment). When decisions have been made about the objectives of the 
system which are inadequately met, the decision-maker might then con- 
cerned with the selection of programs to meet these objectives. Evaluation 
information might be sought relative to the possible impact of various courses 
of action or programs (Program Planning). 

Thus, if one followed through on the full cycle of evaluation in an educa- 
tional system, including the allowance of feedback and recycling, the process 
might be depicted as in Figure 1. 




The evaluation of an instructional program assumes the prior assessment of 
the program or of a larger system, a decision about objectives to be attended 
to, and the selection of programs considered to be appropriate for meeting 
these objectives. That is, the evaluation of an instructional program ordinarily 
begins after the decisions related to need 1 and 2 of the evaluation have 

er|c 
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been made. In evaluating an instructional program, the 
-chieved and the program which it is assumed wM be most 
achieving these objectives are generally considered as Sfiven. the 

evaluation of an instructional program focuses primarily on the last three 

ziccd drc 3 S of cvaluAtion. . £ 

"Where the evaluation task commences with the evaluauon of the instroc- 
donal program, we envisage the necessity for a sub-system assessm^t dcahng 
with the area of concern of the selected instrucuonal program. Thi«. it is 
seen that the evaluation need areas are not necessarily sequential with t e 
steps easily defined. In some instances, moreover, the data coUe^on, analysis, 
and reporting appropriate to a decision might be so e^ to obtain or so in- 
extricably tied to the making of the decision that the decision-makcr and^ 
staff would perform the evaluation themselves. In some uis^nc^, the project 
begins for the evaluator after a number of decisions have akeedy been made. 
Thus, the evaluator might have to attend to only selected evaluation need area^ 
For the sake of convenience, in Figure 2 we have depi'rted a way in which 
the evaluation need areas might be interrelated in the evaluation of instruc- 
tional programs- 




A final explanatory note is in order concemmg the role of the eval^tor m 
this evaluation model. It might be possible to draw the conclusion that the 
evaluator does all things— that he is curriculum designer, administrator, pro- 
gram implementor, test oflScer, budget manager, etc. This is a m«oncepnon. 
We have partially dispelled this notion by commenting carher that what has 
been described in this section is the full range of the evaluation cycle. We are 
describing functions to be performed rather than a role in each evaluation need 
area for a specific individuaL 
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The Evaluertion of 
Occupational Education Programs^ 

Jerome Moss, Jr., University of Minnesota 



The justification for this paper rests upon the validity of three assumptions: 
program evaluation is essential to systematic improvement m educational 
efficienev and effectiveness; an intensification of evaluative activity is highly 
desirable; much of what little has been done to date in the name of program 
evaluation is of questionable usefulness. 

Omsistent with the assumptions, the purposes of the paper are to stimulate 
more productive studies by exploring some of the dimensions of pogrom 
evaluarion an^d to provide a conceptual framework for evaluative efforts m 
vocational, technical, and practical arts educaldon. ^ • i 

The papier will treat the following eight dimensions of evaluation, plus a 
brief section dealing with some implications for action: (a) the impotence 
of program evaluation, <b) some causes of past inactivity in cvaluatioi^ Cc) a 
definition of program evaluation, (d) program outcomes (or evaluative cri- 
teria), <e) program characteristics, <f) two roles of program evaluation, 
(g) evaluation as a part of the educational change process, and (h) some 
research approaches to evaluation. 



THE IMPORTANCE OF PROGRAM EVALUATION 



A basic premise of the piapcr is that program evaluation is impiortanc because 
it provides evidence about the relative merits of programs, thus enablirig edu- 
cators to make more rational decisions about the theories and practice of 
program development and operation. Improved rationality of dccision-malcmg 
is demanded on moral, social, and scientific grounds. • . . 

We have a moral obligation to students to provide them with the best pro- 
grams fiossible. But first, we must be able to identify the best of those available. 



» Published in TcebrUeml Report, September 1968, Re^arch Ccqrdmat ion Unit in Ocm- 
pational Education, University of Minnoioca, and reprinted with the permission of the 
muthor. 
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We must be able to show, for example, whether or not, and in what measure, 
a newly developed program is stiperior to the standard program. 

Wc have a social obligation. As the public investment in vocational, techni- 
cal, and practical arts education rises, it becomes crucial that the money is 
spent with the greatest efficiency for society’s ultimate welfare. 

We have a scientific obligation. Evaluation is an indispensable part of the 
scientific method. Educators hypothesize that programs with certain char- 
actcristics will yield certain outcomes. They develop programs with those 
characteristics and try them out. The measurement of outcomes to confix 
or deny the hypotheses is necessary to produce verifiable knowledge. E'l^lu- 
ation is thus essential to the development of a science of instruction, without 
whicli we shall continue to operate by hunches, authority, tradition, and 

personal experience. . . - • • 

One can be against evaluation, therefore, only by showing that n: is improper 
to seek an answer to the question about the relative merits of educatioi^l 
programs. It is entirely fitting and common to criticize the way in which the 
evahiation is done and the validity of the outcome, but it is hardly ever appro- 
priate to argue against the need for evaluation or the importance or improving 
evaluative techniques. 




SOME CAUSES OF PAST INACTIVITY IN EVALUATION 

In light of its importance, how can the fact be explained that program 
evaluation in vocational, technical, and practical arts education [and rehabili- 
tation short-term training] has been an incidental, casual, and sporadic activ- 
ity? AlS a matter of fact, why have relatively few evaluative studies been 
conducted since the passage of the Vocational Education Act of 19<S3? 

There are philosophical-political reasons. We are faced with critical man 
power and social problems. The public policy decision has been to divert most 
available human and financial resources from both research and evaluation 
into the most visible approaches that can at least sustain the illusion of progress. 
With most public agencies under pressure to produce immediate re^ilts, it is 
no wonder that the need for a good show often overwhelms, scientific objec- 
tivity- it is not surprising that there is little time to revise, throw out, and 
frankly compare. Careers are often at stake. Further, it is somehow ui^ Ameri- 
can to be indefinite and doubtful, or to adopt a try-and-see attitude about any 
proposed public program. Legislators are loathe to provide large ^^sums or 
mouey to tty out: sevciral alternative solutions; wc pick one solution ana 
Obviously, evaluation is done hesitantly, with very grave consequences u^ally 
associated with unfavorable findings. “When ideas that are promising as objects 
of research and honest experimentation [are accepted prernaturely and] give 
birth, through artificial dissemination, to a brood of hy.stcrical fads, thc^ is 
tbe danger that angry reaction w^ill dump out the egg with the shell (Oet- 

tinger, 1968, p. 7^77). _ . _ . - i 

There are personnel reasons ror the relative inactivity. Vocational, techmc^ 
and practical arts education has been handicapped by a shortage of w^eU-tramed 
researchers, and evaluation has not been looked upon (mistakenly from my 
fKsrspective) as a specially rewarding, creative form of research. 

Finally, there have been (and still arc) technical difficulties. The remainder 
of this paper will touch on some of these problems so they will not be enumer- 
ated here. Suffice it to say for the moment that evaluation is a highly complex, 
technically and conceptually demanding activity. Until relatively recently, 
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we lacked the statistical and computational tools necessary to do a reasonable 

From now on, however, the picture must change. The Advisoiy Council 
on Vocational Education has demanded greater efforts at evaluation. Social 
scientists from a wide variety of disciplines are tvuming their attention to the 
assessment of various systems of manpower training. Our social obligation 
for evaluation is being assumed by others, and the results could determine our 
very existence. We must evaluate our own programs using appropriate criteria 
and mcthodolog>’- so that decisions concerning our fumre can be based upon 
data which properly reflects our educational perspectives. 



A DEFINITION OF PROGRAM EVALUATION 



While all vocational, technical, and practical arts educators have some con- 
cept of program evaluation, the literature indicates that the meanings held 
are quite varied. Figure 1, entitled “Major Components of the Evaluative 
System,” is intended to introduce the concept as it is used in this paper. 

The evaluative system starts with students, each of whom differ with respect 
to characteristics which affect their ability to learn at the time they enter the 
program to be evaluated. C ^deuts differ^ for example^ in. relevant aptitudes^ 
achievement, motivation, health, etc., which alone and in interactions, create 
variation in “readiness” for the program. 

The program the students enter has characteristics which provide them 
with educational experiences. It is these characteristics that arc to be evaluated. 
Students are exposed to selected content, which has been organized in specific 
ways, which is presented in certain manners, and to which the students are 
encouraged to respond in particular ways, all under the guidance or manage- 
ment of an instructor with certain characteristics. These ‘ transactions (Stake, 
1967) take p'ace under particular physical and psycho-social environmental 
conditions. 

In addition to the influences of the specific progranri to be evaluated, students 
are inevitably affected by other experiences and conditions in the environment, 
which occur outside of the program, but whose effects might be mistaken for 
outcomes of the program. These experiences can take place at any tune 
after the student enters the program, and before the program outcomes 
measured. For example, students might take a variety of other courses which 
differentially alter their ability to learn the content of the program to be 
evaluated; increases in dependents or extra-curricular experience on a pas- 
time summer or after-school job could change motivation; economic condi- 
tions could alter the availability of particular kinds of jobs after ^aduation; 
military service could result in greatly enhanced occupational skills, etc. 

The interaction of student characteristics, program characteristics, and 
other intervening influences produce actual outcomes. These oincom^ con- 
sist of student or ex-student behaviors, and the effect of those betoviors on 
the school, the community, the economy, society, etc., and other direct con- 
sequences of the program for teachers, administrative patterns, other students, 

etc. 

Finally, the evaluative system contains one or more sets of comparative 
outcomes. These outcomes are anticipated, expected, hoped-for results of the 
program, or they may be the actual outcomes of a differ'^nt program, or the 
outcomes of the same program at different points in time. In all cases, they 
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Fig* !• Alajor Components of the Fvaluative System 



provide the comparative standard by which the relative merits of a given 
program will be judged. 

The components of vbe evaluative system can be utilized to create a more 
formal definition of program evaluation, as follows; 

Program evaloatlon is the process of attributing differences between 
actual and comparative outcomes to program characteristics, under dif- 
ferent conditions of student characteristics and other intervening influ- 
ences, and making a judgment about the value of the program character- 
istics. The process is conducted for the purpose of making more rational 
decisions about programs. 

>Jote that the definition has two important qualities. First, evaluation must 
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be comparative. To report actual outcomes provides only a description of 
what happened- Evaluation requires making a judgment, which in turn ne-- 
cessitates comparing outcomes with some other set of expected or actual 

outcomes. , 

Second, evaluation requires that differences in the outcomes compared muOT 
be attributable to program characteristics or the interaction of program a^ 
student characteristics. Comparing outcomes which do not reflect actual dif- 
ferences in program, but which are due, for example, to differences in students, 
or to other relevant influences, "would be completely misleading. 



PROGRAM OUTCOMES (OR EVALUATION CRITERIA) 

One of the most critical aspects of program evaluation, and the one which 
has thus far probably received the least attention, is the identification and 
measurement of the program outcomes which are to serve as evaluative criteria. 
Everyone affected by evaluation, and that is all educators, must be concerned 
with developing as complete an array of relevant, potential outcomes as pos- 
sible for use by evaluators. What seems needed first is a classification schema, 
matrix, or topology which will help to identify ^ the scope and boundari^^ of 
such an array of outcomes, and to suggest possible ^ criteria for including 
specific outcomes. C^ne such multi-dimensional classification schema, 'with 
some guidelines for filling it with appropriate criteria, is presented below. 



Guidelines 



First, the criteria by which instructional programs are to be evaluated must 
be the outcomes— the products— of instruction. Program characteristics cannot 
be used as evaluative criteria, for, by so doing, we ijssume, rather than prove, 
that those characteristics are good. Ciiven the present state of knowledge, the 
major purpose of evaluation must be to determine which program character- 
istics actually produce the desired outcomes for a certain group of students. 
Almost none of our cherished “principles” of vocational education practice 
have been empirically validated. They have about as much scientific status 
x"£ght no"w^ as old 'wives^ tales. 2V1[any of them, in time, may prove to he peda 
gogically sound. But the point is thaL they remain to be proven. XJntil tbey 
are proven, alleged evaluations based only on tbe presence or absence of 
certain program characteristics are acts of faitb. In fact, they merely serve 
to describe tbe program in terms of variables which w"e presently think arc 



important. 

Second, the matrix of evaluative criteria should include the potential out- 
comes relevant to each of the different philosophies or value systems under 
which vocational, or technical, or practical arts programs might be operated. 
To illustrate, vocational education may be provided primarily to mamtam 
the supply— demand equilibrium of the labor market, or it may be directed 
mainly toward the development of individuals. Different criteria of succe^ 
or at least criteria with different emphases, would be used to reflect accurately 
each philosophical position. Similarly, manpower training programs can be 
thought of principally as relieving poverty or as meeting labor shortages and 
upgrading the labor force. Here too the criteria of success, the expected out- 
comes, would differ. Reduction of unemployment among the poor would 
be a key index of relieving poverty, while the reduction of structural 
employment would be a particularly germane index of an upgraded labor 
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force. The major consideration is that expected outcomes consistent with each 
of these value systems would be included in the complete evaluative criteria 
matrix for vocational programs. 'When two programs of differ^t phitosophi- 
cal bents are compared, both sets of pertinent criteria mu^ be used as the 
basis for comparison in order to provide a clear picture rf each programs 
relative strengths and weaknesses. In fact, no program should be evalu^ed 
only in terms of the objectives stated by i« developers or operators. ^ he 
Question is always what does the program gain and what does it give up. We 
must observe whether the hoped for outcome occurs, while at t^ same tme 
making certain that damaging effects are not offsetting gains. These side- 
effects” of programs are too often completely ignored, and at great p^enttal 
peril to the long-range interests of students and educational institutions. Careful 
examination of the variables related to the optimized program c^racteristics 
and to the anticipated outcomes will help reveal potential side-effects. 

Third, the expected outcomes of educational programs can be stated at 
several levels of specificity. At the most general (macro) level, expected 
outcomes are broadly stated to express directly a -philosophical position. At 
more specific levels, outcomes can be stated as measurable indices or as opera- 
tional definitions of the more generalized statements of expected outcomes. 
At their most specific (micro) level, outcomes can take the form of items on 
an achievement test, a questionnaire, a rating scale,_ a personality inventory , 
and so on, which samples behavior under various stimulus and response con- 
ditions. Obviously, each succeeding level of greater specificity must represent 
a valid sample of the more generally stated ends if measures are to accurately 
reflect basic intents. To illustrate, a general outcome or objective ^r a voca- 
tional program might be to upgrade the quality of the labor force. Cme, inore 
specific, expression of that hoped-for outcome would be to increase the abihty 
of students in preparatory programs to ose^ their skills in a -wide variety of 
situations. A. measurable index of that ability could be the time needed to 
retrain for a related occupation, to some specified level of performance. An 
operational definition of that index nriight be the time it -wifl take to tram 
secretarial graduates of a given ability level from “X” vocation?!! school to 
type 130 three-inch wide justified lines per hour on a 'Varityper with no 
than five errors. The actual performance test and the conditions under which 
it is given and scored constitute one example _of the micro level of the expected 
o^itcome. Of course, many additional indices, operational definitions, and 
items would have to be sampled in order to properly measure whether or not 
the c[uality of the labor force had been upgfraded. ^ i j 

For the purpose of progfraixi operators and evaluators, the profession should 
suggest outcomes expected at both the most g’eneral, philosophical level, and 
at the level of measurable indices. At the present time, this wc uld require the 
development of at least two matrices for vocational and technical programs, 
and two for practical arts programs (see Figure 2). 

Fourth, indices of program outcomes should not only be consistent w^ith 
philosophical positions, but they should also be sensitive to variations in pr^ 
gram characteristics. As far as possible, they should logically and directly 
reflect differences in programs, while remaining relatively independent of 
student characteristics and other intervening, e.g., economic, conditions. The 
function of program evaluation is to attribute differences betrween outcomes 
to program characteristics. Cjross indices, or those which are influenced pri- 
marily by non-educational variables, will make the task of evaluation ei^emely 
diflicult, if not impossible, by hiding, rather than highlighting, the effects of 
program characteristics. 
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Fifth it would greatly facilitate weighing the relative merits of programs 
if it were possible to assign monetary values to program outcomes. Summated 
or per stiident dollar benefits of different programs could then be directly 
compared. If it were also possible to measure the dollar co^of providing each 
program, an index of investment return per program could be obtained. 1 hese 
cos^ben4fit ratios would be most useful measures for comparing the relative 
efficiency of various programs. Unfortunately, economic analysis has greater 
potential than it has current utiUty. A major problem is in placing valid 
monetary values on all relevant outcomes. This restricts the i^efulness of 
economic analysis in that only a limited number of outcomes can be employed 
in the evaluation. One alternative might be to agree to a^ign dollar values to 
various outcomes in a manner consistent with some social philosophy. Such 
an arbitrary technique could serve to make explicit the implicit weightings 
individuals inevitably give to different kinds of outcomes when judging over- 
all program effectiveness. It seems important to reiterate, ^ough, that whether 
eccmomic analyses of programs are limited or complete, they yield data which 
must be interpreted in terms of given student characteristics and other inter- 
vening influences on program outcomes. We must always interpret program 
cost ^d benefit in terms of Johnny, and not Johnny in terms of cost and 

benefit. 

A Multi-Dimensional Classification Schema 

The classification system for expected outcomes should be multi-dimen- 
sional. One likely prospect is a three-dimensional schema consistii^ of a tune 
axis, a target axis, and a type axis. Figure 2, ‘ Matrix of Expected Outcomes, 

depiots tbis possibility. . . ^ ^ ^ 

One axis of the matrix, representing elapsed time since the educational ex- 
perience was provided, is needed to indicate when it is fea^ble to measure 
each expected outcome. One category, for example, might be immediate 
to include all the expected outcomes which can reasonably be measured while 
the student is still in the educational program. Measures of achievement, both 
in the program and in other courses taken concurrently^ drop-out rate, etc. 
belong in the “immediate” category. A second category, intermediate, could 
include outcomes expected to be manifest within some arbitr^ but con- 
venient period, such as four years after leaving the program. Grades ^rned 
in subsequent college or vocational and technical programs, time required to 
find initial employment, the time needed to become satisfactorfly productive 
on the job, etc. are illustrative of “intermediate” outcomes. A thu:d tune cate- 
gory might be “long-range,” and include those outcomes which are feasible 
to measure more than four years after the educational experience. Voting 
record, incom ' and the salience and centraUty of work-oriented values are 
among potential “long-range” outcomes. Some outcomes, of course, can be 
measured in two, or even all three, of the t^e categories. 

It should be recognized that as elapsed time mcreases so does the dimculty 
of data collection and the number of intervening influences, thus complmatmg 
the problem of attributing outcomes to the educational exp^ience. Conse- 
quently, it seems preferable to devote primary attention at this tune to the 
measurement of “immediate” and “intermediate” outcomes. 

One veiw interesting basic issue, which deserves careful study and discussion, 
shall be noted only in passing. To what extent should e.^ucation^ progr^s 
take responsibility for the overt behavior of students (or former students) in 
out-of-school situations in which many of the immediate factors influencmg 
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Fig. 2. Matrix of Expected Outcomes (One Matrix for Each Level of Specificity) 
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that behavior are beyond the control of the school prog’rarn? Conversely, to 
^vhat extent do ^w'e discharge our responsibility to the individual ai.«d society 
by showing the student^s capability to act by his performance under controlled 
conditions? The ansv^ers to these questions will greatly influence our percep- 
tion of appropriate program outcomes. 

A second axis, for target, distinguishes between expected student <and 
former student^ outcomes, and indirect, secondary , or feedback outcomes 
anticipated in other people, agencies or institutions. We might call these 
categories “student"* and “other.” The “student” category would logicaUy 
include an expectation of an “immediate** increase in vocational rnaturity 
among program enrollees; the ** other** category could contain an index or 
“immediate** improvement in parental attitudes to'ward the school. Some addi- 
tional illustrative outcomes in the “other** category are effects upon teacher 
morale, educational administrative patterns, and conimunity support for edu- 
cation. The “other** category might also include indices of the program s 
impact upon firms, occupations, industries, and even upon the commumty, 
the economy, and society as a whole. 

Al third axis is m-cded to distinguish among types of expected outcomes. 
One possible distinction is among educational, psycho-social, and econo^c 
criteria. Educational outcomes are illustrated by the cogmtivc and psycho- 
motor abilities acquired in the program, and by the numbers eff i^rsons re- 
ceiving training in relation to those needing or requestmg it. Reduction m 
the delinquency rate of students, and the attitudes of prospective employers 
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or of other educational institutions toward the program, are examples of 
psycho-social outcomes. E^jonomic criteria include earnings of former students 

and labor cost per unit of production. ^ 

It may be postulated that qualitative and quantitative outcomes also deserve 
separate sub-categories on one of the three suggested axes. However, rather 
than complicate the classification schema fvurther, it seems preferable to include 
both qualitative and quantitative outcomes, where applicable, in every cell- 

The benefits of developing a complete array of relevant, potential outcomes 
for use by evaluators can be summarized as follows: 

a. The process of program development and revision will require con- 
tinuous clarification of extant philosophical positions and exarnination 
of their internal consistency. The array will facilitate this activity, and 
will also point to the kinds of new indices and measurement devic«;s 
that need to be created. 

b. The classification schema will provide the basis for cumulating and 
organizing the results of evaluative studies and for identifying cILrec- 
tions for further efforts. 

c. A less obv' ous but equally significant benefit of an array of expected 
c ccomes Ls to place each evaluative study in proper perspective. That 
is, all evaluation studies will undoubtedly be partial in teimis of the 
limited number of outcomes actually measured. Program operators 
will become more aware of these criterion limitations, and thus be 
better able to make more rational policy and administrative decisioiis 
based upon the results of given studies. 



PROGRAM CHARACTERISTICS 



Program characteristics may be thought of as the sub-systerris, such as the 
teacher, content and content organization, methods and techniques of in stru c- 
tion, facilities, etc., which provide the student with relevant, planned instruc- 
tional experiences. *The effects of all or one of these sub— systems may be the 
subject for evaluation — the independent variables— in any given style. 

A careful description of program characteristics is prerequisite to evalu- 
ation. Many studies fail at this very basic level. The characteristics to be de- 
scribed need both careful definition and ratings using acceptable absolute 
scales should be employed, so that greater interrater reliability is possible, and 
so that the same characteristics can be meaningfully compared across progranw. 

Decs. use e valuation is 3. time-consuming' activity", and eacti study can only 
■provdde data on a limited number , of outcomes, it seems impoitant that_tj^c 
attention of research-evaluators be focused initially upon programs with 
basically different practical and theoretical characteristics < sub-systems). In 
vocational education, for example, we need to turn oiu- attention first to com- 
parisons of (a) vocational, general, and college preparatory programs, (b) in- 
school educational efforts, cooperative programs, and on-the-job training, 
(c) programs that include a large component of “general” education content, 
and those that do not, (d> programs which focus upon relatively specific 
occupational goals, and those that prepare each student for a broad range of 
occupations, <e) programs that integrate occupational content inductively 
rathfis ^j^n deductively, (f) programs that provide for individual differences 
bv content and method, and those that d© C&) programs tvhicb 

eavollment time as an individualized variable, and those that do not, 
(ki3» £ii III! which incorporate information about the psychological and 
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sociological elements of the occupational environment, and those that do not, 

and so forth. , . . ^ 

Each of these comparisons should be made using a variety of content dr wn 
from different kinds of occupations until we learn if and how subject matter 
differences interact with other kinds of program characteristics to affect pro- 
gram outcomes. The nature of the content being taught (learmng task) must, 
until we learn otherwise, be considered an important program characterise 
in the evaluative system. Similarly, the potential variation in outcomes due 
to the effect of individual teacher differences seems to great enough to 
require that some attempt be made to account for their influence. 

Technology is fact making it feasible to individualize instruction. Conse- 
quently, the days when researchers were forced to try to find the one b^ 
method or curriculum for all students are past. The problem now is to identify 
the configurations of student characteristics and program characterise^ C*ti- 
cluding content) which produce desirable outcomes with maximum efficiency. 
It is therefore important that comparisons of program characteristics be made 
so that program outcomes can also be related tr, carefully described differences 

in student characteristics. . _ 

As previously noted, it would be desirable to interpret all program outcomes 
in terms of their dollar value. But it is essential to meaningful evaluation that 
the costs of providing each program be measured. Costs are the inputs, in 
resources and time, utilized by the program to produce outcornes, under 
certain conditions of studer. rs and influences. So long as resources for educa- 
tion remain limited, any realistic evaluation must be concerned with program 
efficiency as ■'veil as effectiveness. Cost per unit outcome (even if unit outcome 
is not in dollars) is such a measure of efficiency. And, as long as final outcomes 
are comoarable, the relative efficiency of different programs can be ascertained 
by using this measure. In computing program cost, social (“other ) as well 
as “individual” costs must be determined, just as social ( other’ ) and indi- 
vidual” program outcomes may be considered benefits. Unfortunately, most 
of our present reporting systems are not producing the kind and quali^ of 
data needed to make complete, accurate program cost estimates. Most spools 
are thereby failing in their responsibility to the public for accountabihty. 



TWO ROLES OF PROGRAM EVALUATION 

As many authors have pointed out, evaluations can be conducted by dif- 
ferent people for many different kinds of reasons. There are, however, two 
primary roles played by evaluation in the process of answering two different 

sets of basic questions. - ^ ^ • 

First v/hat Scriven (1967) has called the “formative role of evaluation^ 
an attempt to aid in the process of developing or improving a pre^ram. It 
answers questions Uke: How well is the program accomplishing what it set 
out to do? How can it be improved? How can program efficiency be maxi- 
mized in terms of accepted program goals? Second, Scriven’s “summative 
role of evaluation is an attempt to estimate an operational program’s overall 
effectiveness; it provides a basis for choosing among programs. It answers 
questions like: Which of these two programs is better for my purposes? What 
would I gain and lose by adopting the new curriculum? 

The differences between these two kinds of questions engender some im- 
portant distinctions in the criteria employed by formative and summatave 
evaluation'-. Formative evaluation derives its comparative base from within 
the same program. Comparisons may be between actual and expected out- 
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comes, present and past outcomes, or between concurrent sets of outcomes 
derived from manipulating certain program characteristics. In the case of 
developmental projects, immediate outcomes are typically used; for improv- 
ing established programs, intermediate outcomes are sometimes employed as 
criteria. Parts of the program or the total program may be the subject for 

evaluation. , i- i_ j w 

On the other hand, summative evaluation can best be accomphshed by 
comparing the outcomes of two or more programs. Since the programs may 
have different intended outputs, the evaluative criteria employed must in- 
clude all of the relevant expected outcomes. The outcomes used are usually 
intermediate, since they frequently provide a more meaningful basis for com- 
paring different programs than do immediate outcomes. 



EVALUATION AS A PART OF 
THE EDUCATIONAL CHANGE PROCESS 

Another way to conceptualize the roles of evaluation is by considering its 
functions in the educational change process. 

It may well be too soon to assess the extent to which recent federal assistance 
for research-related activities has resulted in substantive changes in educa- 
tional programs, but it is already obvious that one extremely significant out- 
come of that intervention has been a concern for systemizing and facilitating 
the very process of educational change. Where formerly it was accepted that 
change would be slow, erratic, and uncoordinated, the availability of rela- 
tively large amounts of money for research and dev ^lopment has instigated 
the study of the educational change process; it has become necessary to decide 
how to invest those funds in order to insure rapid, qualitative improvements 

in education. _ . ^ . 

Figure 3, “Educational Change Model,” illustrates one version of a^general— 
ized abstraction of the change process. It draws heavily upon the prior work 
of Cuba and Clark (1965). ISTote the four places in the model that call for 
evaluative efforts. 

The first place is in the nrocess of inventing and engineering innovative 
educational products. Development must provide for formative evaluation 
as an integral part of the process; developers must know whether their new 
instructional program or device is performing according to expectations; they 
must diagnose early results in order to improve upon efficiency and effective- 
ness. Similarly, developers should be responsible for beginning to accumulate 
data for a summative evaluation so that program operators can eventually 
make more rational decisions about the comparative value of the new versus 
the standard program or device. 

During the diffusion stage, in which innovative programs or devices are 
being demonstrated (on the strength of the results of prior formative evalua- 
tions and preliminary summative evaluation data) , it is inevitable that formative 
evaluation will be continued. It should also be required of developmental 
projects that additional data, including immediate and intermediate outcomes, 
be collected during the demonstration phase for use in making between pro- 
gram comparisons. 

W^hen individual program operators or states elect to adopt a new program 
or device, presumably on the strength of a pertinent sumiriative evaluation, 
there is typically a stage of adaptation and modification which calls for further 
formative evaluation. . i 
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Finally, all operational programs need periodic evaluation. These asse^ 
mcnis should be formative, to improve upon the existing basic program, and 
summative. to test its efficiency and effectiveness against other programs, 
including the new products of later developmental projects. 



SOMt RESEARCH APPROACHES TO EVALUATION 

Formative Evaluation 

Formative evaluations are conducted using a variety of res^ch approach^: 
(a) experts study the content of authenucity, and check the logical consist- 
ency among general purposes, students, content, method, and performa^e 
injures; (b) feedback from program developers, cyalimtors, ^che^ stu- 
dents, former students, etc., based upon direct or indirect ol^rvaaon of 
and/or participation in the program, provides clues for makmg 
changesjfc) pre- and post-tests of a diagnostic nature supply 
infomiation for making potential improvements m the program; (d) 
ments, using program characteristics as independent variables, arc 
employed; (c) developmental, reproduction, and operauonal costs should 
be (although too frequently are not) estimated. 
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Unfortunately, it is usually difficult to know when to be satisfied with the 
results of a formative evaluation— when to stop revising content and method 
and striving for greater efficiency — and when to begin revising measurement 
devices and expected outcomes. Even if performance objectives are stated 
very specifically in advance of program development, our present lack of 
sophistification in setting and stating objectives would lead us either to question 
their validity or to continue striving for greater efficiency in their attainment. 

There is an obvious need for sumniativi; evaluation wiheh provides for 
making objective comparisons among programs. A variety of research ap- 
proaches has been employed in an attempt to satisfy this need, including 
evaluation bj’’ expert- or self- judgments, follow-ups, experiments, quasi-experi- 
ments, and regressional analyses. (Many of these approaches are also useful 
in formative evaluation studies.) 

Expert and Self-Evaluation 

The judgment of program operators, developers, consultants, etc., can, as 
previously indicated, be very helpful in formative evaluation by suggesting 
revisions in program characteristics. Judgments (ratings) of program char- 
acteristics can also be helpful in suminative evaluation by providing “sensitive 
descriptions of the extant programs. But it is thoroughly misleading, and as 
inappropriate use of expertise, to utilize judgments about program character- 
istics as criteria in summative evaluation. Since programs must be developed, 
operated, and perhaps even accredited right now, it is necessary that judgments 
be made about the characteristics which constitute “good” programs. We must 
not, however, delude ourselves into believing that these judgments are neces- 
sarily correct; care must be taken not to compound possible judgment errors. 
Prior judgments must be put to the test of reality whenever programs undergo 
summative evaluations. 



Follow-Ups 



The follow-up is a procedure which gathers data about former students. 
Some times the procedure is employed to secure the opinions of former 
dents about the program. If the investigator is willing to accept these judg- 
ments, they may be used in formative evaluation to suggest program revisions. 
Mor I frequently, the follow-up is used to collect data about the rtatus of 
former students to .serve as program outcome criteria. For instance, inio*ma- 
tion about their work history may be obtained. These data can provide evi- 
dence for use in both formative and summative evaluations, provided that the 
potential differential effects on outcomes of intervening variables and ^udent 
characteristics arc taken into account. For summative evaluations the adjured 
data about former students must be compared with equivalent adjusted uau 
from some alternative program. Too many studies have been reported m 
which a high placement rate (as one program outcome) is assumed ' 3 be v^id 
evidence of a good vocational program, without bothering to cempare that 
rate with some alternative program’s placement rate, or without taking into 
consideration possible differences between programs in student aptitudes and 
in labor demand in the geographical areas concerned. 

Because follow-ups focus upon former student.-, they should not be the 
only means used for collecting data on program outcomes. The impact of 
programs on the school, the industry, the community, etc., are liable to escape 
Q -ttention unless other data collection procedures arc also employed. 
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Experiments 

The experiment is by far the most efficient design for the conduct of 
summative evaluations, and should be used whenever possible. It controls for 
differences in student selection, and many other kinds of extraneous, inter- 
vening variables (history). Some natural opportunities for utilizing true experi- 
ments do present themselves. To illustrate, inadequacy of resources, or the 
need for “tooling-up” time, may necessitate a staged introduction of a new 
program into a school system, thus providing an opportunity for control and 
experimental groups of schools or classes. In other situations, more volunteers 
than can be handled may be recruited, and random assignment provides a 
desirable social and scientific means for forming treatment g. oups. Teachers 
might also be asked to nominate matched pairs of students, each of whom is 
likely to want to cooperate in a new program, and these pairs can then be 
randomly divided into experimental and control groups before actually re- 
questing the experimental group to volunteer. 



Interrupted Time Series 

It is most frequently the case, however, that true experimental designs are 
not feasible to summative evaluation studies. In these situations, the interrupted 
time series (Campbell & Stanley, 1963; Campbell, 1967), a quasi-experimental 
design, should prove extremely useful. The essence of the time series is the 
periodic measurement of some unit and then the introduction of an experi- 
mental change in the unit, the results of which become evident by a discontin- 
uity in subsequent periodic measures. Figure 4 illustr .<'^s four examples of 
this approach. 

Assume that an outcome (or even cost per unit of some outcome) of a one- 
year vocational program, such as percent ot immediate placement in a related 
occupation, is measured each year for succeeding classes so that measures are 
available for 1966, 1967, and 1968. Then, in 1968-69, a major revision is intro- 
duced into the program. Percent of immediate placement in a related occupa- 
tion is again measured for the 1969, 19670, and 1971 classes. 

Was the program revision successful? If time series A or B, in Figure 4, 
occurred, then it appears that the revision had no effect, but, if examples C or D 
were found, it would appear that the new program had a very desirable effect 
on the measured outcome. In fact, if the same criterion measure was used in 
all years, the only two plausible alternative explanations of the changes in time 
scries c’and D, other than the revised program, would be abrupt differences 
in student characteristics and/or other historical, c.g., socio-economic, in- 
fluences after 1968. If data is available for all years on students and on germane 
historical conditions, these alternative explanations of the jump in outcome 
can cither be eliminated, or regrecsional analyses can be used to adjust the 
outcomes for their differences. Should comparable data also be available for 
the same years from other programs in the same labor market area, it is po^ible 
to begin to use cross-program comparisons (multiple time-series) to eliminate 
some historical conditions as possible sources of internal invalidity. 

It is obvious that, to be considered significant, the change in measured out- 
come should exceed the fluctuation normally expected in the time series. The 
logical test of the hypotheses of no change would therefore be between some 
extrapoiated-cxpected outcome of the time series which preceded the experi- 
mental intrusion and the actual outcome after the experimental treatment. If 
continuous improvement, as in example D, is hypothesized, Campbell (1967) 
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Fig- 4. Interrupted Time Scries 
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suggests that the appropriate test is between the slopes of the time series before 
and after the experimental treatment (in addition to a test of difference in 
expected and actual intercepts). Glass and Afaguire (1968) have recently 
investigated the applications of certain statistical models for analyzing the 
change in level of performance- 



Regressional Analysis 



Regressional analysis has recently be^n to be employed in formative and 
summative evaluation studies as a statistical techmque for accounting for 
variation in program outcomes that are related to differences in student char— 
Q eristics and other intervening variables. 
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The evaluator proceeds by gather iujr data on outcomes, student character- 
istics, and intervening variables from rli« programs to be evaluated. Assuming 
two programs are being compared in a suiijmwtive evaluation, a full multiple 
linear equation of the general form shovvn as 1 ) in Figure 5 is developed tor 
the combined programs, and is determined. 

Then, as also shown in Figure 5, Pi and are deliberately made equal to 
O and a new Rr* is computed from the restricted equation. The significance 
of the difference between Ri* and Rr* can be tested by the F statistic. A 
significant reduction indicates that program membership does account a 

significant amount of the variation in program outcome, independent of the 
student characteristics and intervening variables used in the equation. It means, 
in effect, that after accounting for differences in program outcomes due to 
variations in certain student characteristics and intervening variables the kind 
of program provided still has a significant relationship to outcomes- 

Other general equations are available to test for ii:teractions between pro- 
gram and student characteristics and between program and other influences. 
In addition, second and third degree polynomials can describe curvihnear 
functions. Canonical correlations can relate combinations of independent 
variables to combinations of dependent program outcome variables, all in one 



equation. . . , 

Regressional analysis is therefore a most_ useful, almo^ adispensable 
in evaluation. There are, however, certain limitations of cue technique which 
should be made explicit. First, a fairly large sample of students, which increas^ 
•with the number of independent variables used, is needed to obtain reliable 
results- Second, unless the analysis can be cornbined with a true experimental 
design, or unless R(* = 1, the relationships yielded by the equations are not 
necessarily casual. The relationships found are based entirely upon the variables 
used in the equation; the use of additional variables could change the relation- 
ships found. Third, the relationships revealed arc peculiar to the sample 
or to other groups of students of which the sample is representative. The 
sample, for example, might happen to consist of students with quite sunilar IQ 
scores. IQ would then probably not reveal a high relationship to achievement 
as a program outcome because of its restricted range. In another more hetero- 
geneous sample, IQ could, and probably would, be found to have a strong 
relationship to achievement. 



1) Y' -- + b^Pi + bgPji + bgS + b^I, which yields 

where Y' is a predicted program outcome 
is a constant <computed) 

is membership or not in one of cne two programs 
is membership or noc in the second program 
is values from student characteristic <s> 
is values from intervening variable (s) 
are partial correlation coefficicnis (computed) 
is the percent of vl. iation in Y explained by the full equation 

?) Setting Px = Pa = O: 

Y' = a 2 + bflS -b b^I, which yields 



Pi 

Pa 

S 

I 
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Fig. 5. Regression Analysis Model 
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IMPLSCATIONS FOR ACTION 

It is quite urgent that states begin to plan for the p^iodic collection of <^ta 
which Ire necessary to evaluate existing programs. This paper has Pr°^‘ded 
Tm^iinlle and guidelines for planning and conducting those evaluat^r^, It 
has recommended that the data to be collected include measures of (a) studv nt 
characteristics, (b) program characteristics and program costs, (c) mte^e 
var-iables and (d) program outcomes. But it remains for states to decide 

specific fo be utiUzed, the means for measuring them, and the 

techniques for collecting, storing, and retrieving the measures. 

TudSments about the potential fruitfulness of relevant variables and their 
miasuf^?nt mui be made, based upon presently available ^now edge 
expert opinion. Even though some decisions may later prove to be 
the “evaluative system” should be established just as soon as is possible 
conduct careful reviews and to make considered judgments. ^ 

wide variety of specialists, within and outside of education, must be an integral 
part of th?decision-making process. Researches can then assmne the long- 
term task of seeking new knowledge to improve the system. -i-... 

And the federal government has a pivotal role to play. It has a rMponsib^ty 
to stimulate activitv, to facilitate communication among ^ates, and to pro id 
loSXdon tlSoukhout the development of the eystem. F.i^lly “ ’S"SedXr 
that some reasonable minimum amount of i^ormation to be coUected for 
descriptive and evaluative purposes is comparable among the states. 
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EVALUATION OF TRAINING^ 



Donald L. Kirkpatrick, University of Wisconsin, Milwaukee 



An effective training director will make an effort to evaluate all of his 
training activities. The success of these efforts depends to a large extent on a 
clear understanding of just what “evaluation” means. This chapter will attempt 
to accomplish two objectives: (1) to clarify the meaning of evaluation, and 
(2 ) to suggest techniques for conducting the evaluation. 

These objectives will be related to “in-house” classroom programs, one of 
the most common forms of training. Many of the principles and procedures 
can be applied to all kinds of training activities such as performance review, 
participation in outside programs, programmed instruction, and the readmg of 
selected books. 

The following quotation from Daniel M. Goodacre is most appropriate 
as an introduction: 

Managers, needless to say, expect their manufacturing and sales depart- 
ments to yield a good return and will go to great lengths to hnd out 
whether they have done so. "When it comes to training, however, they 
mav expect the return — but rarely do they make a like effort to measure 
the actual results. Fortunately for those in charge of training programs, 
this philanthropic attitude has come to be taken for granted. _ 1 here is 
certainly no guarantee, howe^/er, that it will continue, and training^ oirec- 
tors might be well-advised to take the initiative and evaluate their pro- 
grams before the day of reckoning’ arrives. 



EVALUATION CLARIFIED 

TMearly everyone would agree that a definition of evaluation would be “to 
determine the effectiveness of a training program.” But this has little meamng 
until we ans wer the question, “In terms of what?” We know that evaluation is 

1 Published in R. L. CraiR and L. R. Bictel (Eds.) Training and 
New York; McGraw-HiU Book Company, 1967, and reprmted with the permission of the 

author and publisher. jm 
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needed in order to improve future programs and to eliminate those programs 
that are ineffective. The problem is how to begin. ... t , 

Evaluation changes from a complicated elusive generality into clear and 
achievable goals if we break it down inr.o logical steps. These steps can be 
defined as follows: 

Step I— Reaction. How well did the conferees like the program? 

Step 2— Learning. What principles, facts and techniques were learned? 

Step 3 — Hehavior. What changes in job behavior resul 'ed from the pro- 
gram? 

Step ‘^Results. What were the tangible results of the program in terms 
of reduced cost, improved quality, improved quantity, 
etc? 

With this clarification of the meaning of evaluation, training men can 
begin to pinpoint their eitorts at evaluation. They better reahze what 
are doing and they recognize the limited interpretations and conclusions that 
can be drawn from their findings. As they become more experienced and 
sophisticated in evaluation design and procedures, they slowly begin to obtain 
more meaningful results on which future training can be based. 

These four steps will now be defined in detail with examples and suggested 
guideposts. It is important to stress that the described procedure and tech- 
niques can be used in almost any organization. It is also important to stress 
the results from one organization cannot be used in another orgamzation. 
Obviously, there arc many factors that would influence the results. These 
variables include the group, the conference leader, and the approach to the 
subject. 

STEP 1— REACTION 

Reaction may best be defined as how well the trainees liked a pa: l:icular 
training program. Evaluating in terms of reaction is the same as measuring the 
feelings of the conferees. It is important to emphasize that it does not include 
a measurement of any learning that takes place. Because reaction is so easy 
to measure, many training directors do it. 



Guides for Evaluating Reaction 

1 . Determine what you want to find out. 

2. Use a written comment sheet covering those items determined in step 
one above. 

3. Design the form so that the reactions can be tabulated and quantified. 

4. Obtain honest reactions by making the forms ananymous. 

5. Allow the conferees to write in additional comments not covered by the 
questions that "were designed to be tabulated and quantified. 

The comment sheet shown in Fig. 1 was used to meas’^re reacti^ at an 
ASTD summer institute that was planned and coordinated by the staff of tlie 
IVlanagement Institute of the University of Wisconsin. ^ ^ . 

Those who planned this ASTD program were interested m reactions to 
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subject, technique (lecture versus discussion), end the perfomance of the 
conference leader. Therefore, the form was designed accordingly. So the 
reactions could be readily tabulated and quantified, the conferees were asked 

to place a check in the appropriate spaces. , . , • r i 

In question 3 concerning the leader, it was .elt that a more meamngfifi 
rating would be given the leader if the conferees considered iterns A through 
G before checking the overall rating. This question was designed to prevent a 
conference leader’s personality from dominating group reaction. 

Question 4 allowed the conferees to suggest any improvements that came 
to mind. The optional signature was used so that follow-up discussions with 
conferees could be done. In this ASTD summer institute, about half of the 
conferees signed their names. With this type of group, the optional signature 
did not affect the honesty of their answers, in all probability. It is strongly 
suggested that unsigned sheets be used irx most in-house meetings, however. 

This ASXO reaction sheet was used at the conclusion of every session in the 
institute program. Therefore, the conferees rated each conference leader for 
his contribution to the program. In many internal training programs, a senes 
of meetings will be held and the reaction sheet will not be used until the end 
of the last session. This is especially tiue when one conducte 

the entire program. In this case, a comment sheet like the ASTD one might 
be adapted to the sicuation and modifications made. ^ ^ _ 

A very practical suggestion came from Richard Johnson of Con-Gas 
Service. He suggested that the comment sheets be given to the enrollees befo^ 
the program is over so that the suggestions can be used in improving the last 
section of the training program. For example, where a training program 
consists of a series of nine sessions, the comment sheet should^ be given to 
conferees at the end of the third session. Their comments and suggestions 
should be taken into consideration to make the last six sessions more effective. 

How to Supplement the Evaluation of the Conferees 

It has been emphasize^ chat the form should be designed so that tabulations 
can be readily made. In this writer’s opinion, too many comment sheets are 
still being used in which the conferees are asked to write in their answers to 
questions. Using a for-n of this kind, it becomes very difficult to summarize 
comments and to det 'rie patterns of reaction. 

At the Management Institute of the University of Wisconsin, every session 
is evaluated in terms of the reactions of the conferees. This has been done for 
more than 15 years. Occasionally the coordinator of the program felt that the 
group reaction was not a fair evaluation of the effectiveness of the program. 
Sometimes the staff men felt that the conlrerence leader s personahty made 
such an impression on the group that he received a very high rating. In other 
sessions, the coordinator felt that the conference kader received a low ratmg 
because he did not have a dynamic personality. Therefore, the Managem^t 
Institute adopted a procedure by which every conference leader is rated by 
the coordinator as well as by the group. A form similar to Fig. 2 can be used. 

This procedure in which the coordinator of the program also evaluates each 
conference leader was also used in an ASTD .summer institute. It was found 
that a coordinator’s rating was usually close to the group s rating, but m some 
instances it varied considerably. In selecting and orienting future conference 
leaders for ASTD institutes, both of the evaluations are taken into consider- 
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A STD Institute 

Leader Subject 

Date 

1. Was the Subject Pertinent to Your Needs and Interests? 

□ No □ To Some Extent D Very Much So 

2. Idow Was the Ratio of L-ccture to Oiscussion? 

rn Too Much Eecmre □ O.K. □ Too Much Oiscussion 



3. Rate the Leader on the Following: 





Excellent 


Very 

Good 


Good 


Fair 


Poor 


A. How well did he state objectives." 












B. How well did he keep the session 
alive and interesting? 












C. How well did he use the blackboard, 
charts, and other aids? 












E>. How well did he summariz?" 
during the session? 












E. How well did he maintain a 
friendly and helpful manner? 












F. How well did he illustrate and 
clarify the points? 












G- How was his summary at the 
close of the session? 













What is Your Overall Rating of the Leader? 

□ Excellent □ Very Good □ Good a Fair □ Poor 



4. What Would Have Made the Session More Effective? 



Signature (optional) 



Fig. 1. Rating Chart 

It is suggested that the training director in each company consider this 
approach- A trained observer such as the training director or another qualified 
person "would fill out an evaluation form independent of the group s reactions. 
A comparison of the two would give the best indication of the effectiveness 
of the program. 
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Coordinator’s Rating of Leader 

Date 

Rating Name of Leader,^-.^ Subject 





Very 
Much So 


To Some 
Extent 


No 


A. PREPARATION 

1. Did he prepare for the meeting? 








2. Was his preparation geared to the group? 








B. CONDUCTING 

1. Did he read his material? 








2. Did he hold the interest of the group? 








3. W’as he enthusiastic- dynamic? 








4. Did he use visual aids? 








5* Did he present his material clearly? 








6. Did he help the group apply the material? 








7. Did he adequately cover the subject? 








8. Did he summarize during conference 
and at end? 








9. Did he involve the group? 






* 



C. CONSXRUCmVE COMMENTS 

1. What would you suggest to improve future sessions? 



U. POTENTIAL 

X. With proper coaching what would be tlic highest rating he could achieve? 
E- ADDITIONAL COMMENTS 



Fig, 2. Leader Rating Sheet 



Measuring Reactions to Outside Training Programs 

The forms and suggestions that have been described above will appiy best 
to an internal training program. Since many companies send their management 
people to outside training programs at universities, American Management 
Association, National Industrial Conference Board, etc., it is suggested that 
rhe reaction of each person attending such a program be measured. Lowell 
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Results of University of "Vi^isconsin 
Management Institute Progi’am Questionnaire 



Key Programs 

A Modern Leadership for Middle Management 

B Supervisors’ Leadership in Cost Control 

C Developing Supervisory Skills 

D Human Relations f or F oremen & Supervisors 

E Leadership and Growth 

F Creative Thinking for Supervisors 

G Human Relations for New F oremen 

TT Totals 

A B 

Quctionnaircs returned: ^ ^ 

B^esponses: 

1. I thought the program was: 

A. Very well organized and helpful 3 3 

B. It was of some value 

C. It was poorly organized and a waste of time 

2. In reference to the subject content: 

A. It was all theory and of little practical va&ue 

B. It was both theoretical and practical 

C. It was very practical and useful 

Concerning the quality of the instruction: 

A. The instruction was excellent 

B. The instruction was average 

C. The instruction was of poor quality 



C 

5 



D 

11 



11 



E 

5 



F 

1 



G 

1 



JL 

29 



29 



3 

O 



2 

1 



2 

3 

4 
1 



3 

9 

11 



1 

4 

4 

1 



1 

1 



11 

1 19 

1 26 
2 



Fig- 3. Oscar Mayer & Co- Evaluation Form 

Reed, former training director of the Oscar Mayer & Company of Madis^, 
Wisconsin, used the form in Fig. 3 for evaluating the reaction to the Umversity 

of Wisconsin Management istitute program. j * „i, _ 

In this situation, Oscar May er & Company was not interested m the reaction 
to specific leaders. They were interested in reaction to the overall program 
to determine whether or not to send other foremen and supervisors. In other 
words, this particular questionnaire was designed to fit the need of Oscar 

^ AnothS^SimpSiy used the form in Fig. 4 to evaluate the reaction of their 
managers who attend an outside program. 



Conclusions about Reaction 

The first step in the evaluation process is to measure the reactions to training 
programs. It is important to determine how people feel about the programs 
rfiey attend. Decisions by top management are frequently made on the basK 
of one or two comments they receive from people who have attended. A 
supervisory training program may be canceled t^cause one superintendent 
told the plant manager that “this program is for the burds. 
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An Insurance Company Study 

In a recent letter, S. W. Schallcrt of the Farmers Mutual Insurance Com- 
pany of Madison, Wisconsin, reported to me on an evaluation he had made. 
A number of their claims adjustors were enrolled in the Vale Technical Insti- 
tute of Blairsvillc, Pennsylvania. The purpose of the three-week course was 
to improve the ability of adjustors to estimate and appraise automobile physical 
da. nacfc. 

The specific technique used by Schallcrt was to have the adjustors keep 
trade of their saving’s for approximately six months after returning from Vale. 
These savings were the difference between the estimate of damage by garages 
and the estimate of damage by the claims adjustors who had been trained at 
Vale. Where the final cost of the adjustment was the same as the estimate made 
by the Farmers Mutual man, this was considered the savings. 

In other words, the purpose of the training was to prepare the adjustors to 
make estimates wliich they could justify and sell. Actual dollars and cents 
figures could then be used to determine whether or not the cost of sending 
these adjustors to Vale was justified. 

A Cost Reduction Institute 

Several years ago, two graduate students at the University of Wisconsin 
attempted to measure the results of a **Cost Reducti on Institute*’ conducted 
by the Management Institute of the university. Two techniques were used. 
The first was to conduct depth interview^ with some of the supervisors who 
had attended the course and with their immediate superiors. The other tccra- 
nique was to mail questionnaires to the remaining cnrollees and to their super- 
visors. Following is a brief summary of that study: 



DEPTH INTERVIEWS 



Interview with Traine.^s 

1. FTnvc you been able to reduce costs in the few weeks that you have been 
back on the job? 



Replies: 



1 3 men 
3 men 
2 men 
1 man 



es 



No 

Noncommittal or evasive 
Failed to ans^ver 



2. How? What were the estimated savings? 

Different types of replies indicated that the thirteen people who said they bad 
made cost reduertions had done so in different area;.* But their ideas stemmed 
directly from the program, according to these train^^cs. 



Interview of Superiors 

Blight of the cost reduction actions described by the trainees were confirmed 
by the immediate superior and these superiors estimated total savings to be 
from SI 5,000 to S2 1,000 per year* The specific ideas that were used were 
described by superiors as well as by the trainees* 
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MAILED QUESTIONNAIRES 

T®*’*"'* *=rainces who were not contacted per- 

questionnaire were not nearly as specific and useful 
^ u? ^ obtained by personal interview. The study concluded that it is 

P^r^jonal interview rather than a questionnaire to 
measure this kind of program. 

Measuring Organizational Performance 

sophi'^ticated and penetrating article related to evaluation was 
'U rittcn by Rensis Likert. It appeared in the March- April, 1958 , issue of the 
Harvard Bustne^ Review. It shows how changes in productivity can be 
measured on a before-and~after basis. Two different types of groups were 
used. The first v.'as a poup of supervisors trained in using a democratic kind 
®‘*^^®^*-** *" which decision making involved the participative technique. 
The supervisor in the other group were trained co make their own decisions 
and not as ^ subordinates for sug’g’cstions. 

as results in terms of productivity, such factors 

M loyalty, attitudes, interest, and work involvement were also measured. 
Where both traimng programs resulted in positive changes in productivity, 

human'SSL'facto^.'' ■" better feelings, attitudes, and other 

The article described another excellent study from the University of Michi- 
Likert concluded by saying that “industry needs more adequate 
measures of org' --;',ational performance than it is now getting.” 

Conclusions about Results 

The evaluation of gaining programs in terms of “results” is progre.ssing at 
as rh^ slow rate. Where the objectives of training programs are as specific 
as the reducuon of accidents, the reduction of grievances, and the reduction 
rcsear^’e^^lf * number of attempts have been made. In a few of them, the 
r at^mpted to segregate factors other than training which 

might ha^^ had an effect. In most cases, the measure on a before-and-after b.asis 

have Se^ hXe^^^^ training even though other factors might 

Studies like those of Likert attempt to penetrate the difficulties encoun*^ered 
in measuring such programs as human relations, decision making, and the like 
In the years to come, we will see more efforts along this direction, and even- 

teAns ^ human relations training, for example, in 

terms of dollars and cents. At the present time, however, our research tech- 
mques arc not adequate. lcco 



SUMMARY 

purpose of this chapter is to stimulate training people to take a pene- 
^ating look at evaluation. Their own future and the future of their orograms 

c” ability to evaluate and use evaluatio*n r£ults. 

objecuve has been to clarify the meaning of evaluation. By break- 
mg It down into reaction, learning, behavior, and results, the training person 
do somrthing about it and can gradually progress from a simple 
subjective reaction sheet to a research design that measures tangible resuite. 
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Articles on evaluation will continue to appear in the Training and Develop- 
mein Journal and other magrazines. Some of these articles are well wortli 
studying^ because they describe effective principles, procedures, and methods 
^ evaluation. One example of an excellent article is “Performance Oriented 
T raimng; Results Measurement and Follow-up,” by Lei'da and Cross, which 
appeared in the August, 1962 , Journal of the ASTD. Another fine article 
appeared in the October, 1962 , Journal of the ASTD. It was entitled “The 
Development and Kvaluation of a Programmed Training Course for Sales 
personnel, by J. S. Abma. These articles arc worthwhile because they provide 
helpful principles and approaches which other training people can borrovi'. 
Likewise, many of the evaluation articles that continue to appear aren’t worth 
the tune it takes to read them. A list of other worthwhile reading appears ar 
trie end of this chapter under “References.” 

TTWs brief chapter has not provided the answers to the traininc- director’s 
problem of evaluation. It has attempted to provide an understandings of priiici- 
pies and methods. Better understanding will come from continued studv of 
ne-.v ppncipl'^s and methods that are described in articles written in profes- 
sional journals. iVeedless to say^ skill in using proper et'a^uation methods can 
only come from p; actice. 
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ON-GOING PROGRAM EVALUATION^ 

Bryan Smith, University of Florida 



jytost Research and Training: Centers are structured on traditional or^ani- 
^tion theory. This theory, which is hased on a static concept of man, is often 
incompatible with the role expectations of the workers in these organizations 
and leads to functional inefficiency and ineffectiveness ironically, these are 
the major undesirable qualities that originally promptc - - ^ 

traditional model. A new organizational model which 
namic concept of man is needed. Both governmental a^ 
sector in our free enterprise system have utilized new 
to meet this need. These techniques include Contract 
cording Analysis and ZVlanagemcnt Implcm 

Control Technique (/wP^CT); I.east Cost F^timatin 
tern and Program Evaluation and Review Tc 

^his^latte^r method "which is compatible with the structure and function of Re- 
habilitation Research and Training Centers. Because of this potential, the 
PERT process is recommended as a procedure to be utilized as an on-going 
program evaluation tool. 



he creation of the 
zes a modem dy- 
and the private 
agerial technique^ 
L Requirements Re- 
lation, Planning and 
ind Scheduling Sys- 
inique (PERT). It is 



THE NEED FOR PERT 

Traditional organization theory was built on a combination of an accounting 
and industrial engineering model. The responsibilities of the organization are 
rationally assigned to a group of boxes spread about in a structure resembHng 
a farnily tree. It utiUzes a balance system of authority and responsibility. Each 
box is ve^ed with a certain amount of authority and simultaneously receives 
responsibility for productivity. 

This balance is truly unique to this model. In no other part of life is authority 
equivalent to responsibility. Any arrangement whereby responsibilities are 
prccLscly and explicitly defined and limited cannot be regarded as efficient. 
It is expensive to the organization when a worker does exactly what he is told 
and no more. 

1 Paper printed at the seminar eodcled, “Stxatesies for the Evaluation of Short-Term 
Traimna m Rehabilitation,” February 1969, Rehabilitation Research and Training Center 
m Mental Retardation, University of Oregon. 
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Output is the rosponsibilitx of the hierarchical line. The staff is to advise 
support, and provide expert assistance to this line. They only have authority 
wirnin thex^ own staff groups. T lieoreticalJy', if a balance is maintained between 
respon^bility and authority, the staff which lacks authority has no responsi- 
bihty. To equalize responsibility and authority demands that these commodi- 
ties be measured— a task which defies logic. 

Another characteristic of c.assical organizational theory is that it maximizes 
neatness and coiitroL Breaking- up the org-inizatioual goals into objectives and 
placing them into appropriate boxes gives the impression that one knows what 
ought to be done everywhere and that there is always an overview of the 
'which a box is responsible. When something goes v*T^ng, reper- 
cussions can be heard all the way down the organiz-ational structure. TThere 
K special emphasis on error, its detection, and correction after it has happened. 
1 nc standard organization is set up so t^int everyone has an assigned task As 
soon as the task is accomplished, it is possible 1:0 discover the error rnake 
appropriate adjustments, and then see that it is done. The upper levels’ of the 
hierarchy are relieved that they have made it through another day without 
anyt.iing going seriously wrong. These people at the higher levels always 
have a check to be sure that, contrary to what they were afraid of, nothing 
s^ious had gone wrong in the system. It docs not provide the symmetrical 
cnecK as to whether anything serious had gone right. 

Another characteristic of the traditional model is that growth or develop- 
ment in the individual is discouraged since it will upset the system. The 
worker obtained a job that was appropriate for him. If he develops new skills, 
fie will either want to reach out and do more or work less. Both have deleterious 
effects upon his and others’ morale. The stability of workers, both in mini- 
mizing turnover and change within the individual, is desired. 

These are a few of the characteristics which describe the traditional ap- 
I^oach to organizational structure. What is the nature of man who fits into 
this system? What assumptions are made about the labor force from which 
we draw our v. orkers into research and training centers? Within the tradi- 
tional stracture there are procedures and systems which imply to the worker 
that he is lazy, shortsighted, liable to make mistakes, selfish, and a little dis- 
honest. Since the system does not let him see all the factors involved, his 
judgment:, ii lie lias any, is apt to be poor. 

lukert (Haire, 1959) analyzed worker attitudes at various levels in »-he 
. *P*^^^*" . order. He asked job foremen to rate the importance of certain 

^b qualities as tliey viewed them for themselves, and for the ^v'o^kers at lev-els 
below and above them. The general foremen were asked to complete the same 
rating form for themselves and as they thought the foremen and the superin- 
tendent would do it. In each case, no matter what the level of the respondent 
ms job and his superiors demanded admirable personal qualities, such as 
iriitiative. His subordinates did not. 

^ oew decentralized organizational mode! were designed to replace the 
traditional hierarchial ladder, it should maximize participation and utilize 
worker initiative by creating organizational equality for everyone. In seeking 
initiative, there would be an atmosphere in which there is freedom to make 
mistakes, to communicate, to make new approaches, to learn, to grow, and 
to develop without someone anticipating failui'e. The growth and develop- 
ment of the workers are the primary goals. Through such a model a Haw- 

Effect will be injected, creating an improved organization and product 
produced by it. 
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PROGRAM IMPLEMENTATION THROUGH PERT 

PE^RT is a proven management tool which enables an organization to citti- 

optimum use of staff time, simplify budget prepa- 
ration, give advance warning of impending bottlenecks, and minimize Td 
mmistrative overload It offers a number of indghts into ?he oTSi^ization’^ 

provides the freedom for all workers to explore numerous 
alternatives for the accomplishment of program goals. Tiumerous 

United States Navy in 1958 to coordinate 
manufacture, and assembk.- of the numerous componen-s 
H ^ operational Polaris missile. It was later ifsed bv 

m ^L*^*^*^ services and numerous industrial and goverii 

tliat consists of plotting the network of activities 

^ T ^ outcome or objective. The Critical Path Method fCFykf ) 

was developed about the same time by DuPont and is a similar method of 
^ sequence the steps that must be taken to reach an objective 

Both techniques are methods for rational planning and seek to coordinate a 

f^^^c^fi^out -SmT^-rh and efforts into a plan designed to achieve a 

specihc outcome. They require the listing and ordering of events and activities 

acrfv'ir""^ must be done, how long it will take, and the relationship of 

can be mad^bSed accomplishment of the program. Calcularions 

^ based on time estimates to show the critical timing of the activi- 
oh^' program is started, the probability of achieving the program 

objective in the time allowed can be calculated and, if necessarv infomation 

"p£^Va?d system to alter the sequences. 

provide a visual estimate of what needs to be done, hovz 
long It will take to do it, what kind of activities must be carried out when 
wMch’ by^^hom. Xhe main advantage is the provision of a mechTnism 
aeHvS-^ estimates of the time necessary to accomplish a sequence of 

w\^t ^^^W’^^^ieous y allows for rapid and clear communication be- 

relarioncl ‘ possiblc for each person to see his and his co-workers’ 

relauonship to the success or failure of the total process. worKers 



PERT Steps 

(Step 1). The outcome must be tangible and 
after thic^ cornpletion of an activity. Xhere is nothing more to be done 
ion takes place. It should not include any processes or previous 

nS resources'^""" "" milestone. It^ consumes neither 

SrocSsed cards of events are: meeting held, report mailed, data 

^ocessed, cards punched. Xhey are occurrences which may happen during 
the course of the program and are not subject to evaluation. ^ 

Uetemnnitig precedent events (Step 2). All the events which precede the 
final event must be recorded. Xhis listing does not have to be irf a serial or 
chronological order. For example, if the final event is: center opened the 
precedent events might be: staff hired, materials ordered, equipment installed, 

be"2^de^e7 xtfthe «^hey should 

whC-h ^ ^ ^ ® number indicating the order in 

which these events must occur. For example, before hiring the Center’s staff 

It IS necessary for the workers to be accepted by the Board of Directors Npw 
events may have to be added to the list. Directors. New 
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Tabic I. 

Ordcriruj’ Preceding” Events 
Sequence Number Event 

1 Organizational meeting held 

2 IVlotel contacted for seminar space 

3 List of lecturers completed 

4 Honorarium approved by administration 

5 Eight lecturers recommended 

6 Syllabus for each seminar completed 

7 Syllabi submitted for approval 

8 List of trainees obtained 

9 Ooordination meeting held for lectures 

10 Trainees enrolled 

1 1 L*ecrurers assigned to rooms 

12 Seminar started 



Preceding Event 



0 

1 
1 



5 



3 

4 
6 
1 



7^10,11 

8 



2.4 

9 



Once the events have been listed in sequential order and the preceding 
events have been determined, they can be plotted into a flow chart. This chart 
consists of a system of paths of sequential events contributing to the final 
program objective (for description of events see Table 1). The paths are 
parallel in the time sequence and each represents a sequence of time. The 
length of the path, which represents the activities between two events, are 
not drawn to scale. 




Each event must be followed by a successor event. Each successor event 
must have at least one predecessor event- There should not be any loops, 
crosses, or dangles such as those illustrated in Pigure 2. 

Deter7ni72ation of activity time (Step 4). Certain activities must take place 
between each pair of events. These activities must be listed, and the amount 
of time that each would take must be estimated. In making the time estimate 
(te) three T^redictions are used: the minimum time (a) is the minimum possible 
period of time in which the activity can be accomplished. The best time (m) 
is the best estimate of the period of time in which the activity can be accom- 
plished. The maximum time (b) is the maximum possible period of time it 
would take to accomplish the activity. These activity times are then placed 

into the following formula to calculate the time estimate: to ^ +• 4m -f- b. 

Allocation of priority activities (Step 5). The activity time estimate is 
added to the flow chart on the appropriate line ber'veen the two events. Once 
the. time estimates are on the flow charts, it is possible to determine which 
activities are priority activities, and which line on the chart represents the 
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Fig. 2. 





critical path. ’ The critical path is the longest time from the first event to 

therefore, the quickest possible time the program can be 
completed. These are a senes of events which have other events dependent 
on thern; thu^ need priority attention. Resources and personnel can b- 

allocated on the basis of the flow chart. 
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T he flow chart will reveal how conciirrent activities can exist and how 
more than one may be taking place simultaneousl 3 ^. These parallels can be 
conducted at the same time by one or more persons in the organisation. 

The tin?e estimates max vary markedly bet^vceri minimum and maximum. 
The wider the separation between these estimates, the greater the uncertainty 
associated with the activity. This uncertainty can be expressed by the statis- 
tical tool known as variance It is descriptive of the uncertainty ass^jciated 

with the distribution. If the uncertainty is small, the variance will be small. It 
shows how precise the activity time estimate is. 

Variance can be calculated by the following formula; 

cr2 = (b — a)2 

< 2 ) 

The expected cornpletion time (Ttq) should also be calculated and added 
to the flow chart. It is the earliest possible time that the event can be reached 
and is calculated by adding the t© of the activities leading to each event. The 
latest allowable completion time ( X ji,') can now be calculated from the Tb 
by subtmeting the te. TPhe values of all the TTi^ are calculated for each event 
and included on. the flow chart below each event. In this computation start 
with the last event an I work back toward the first one by subtracting the te 
from the value of the T e for the successor event. 

From the values of the T® and Tx, the amount of slack time (To) can be 
calculated. Slack time is the amount of time in excess of the te that is available 
to complete the activity. The slack of an event is T^ — Te. Therefore, if you 
are given 30 days to complete an activity and can actually do it in 25 days, 
you have a 5-day grace period. Slack time is important in aiialyzing the com- 
pleted project when time c f completion is a vital factor. The flew chart can 
now reveal those areas which have an excess of manpower or ma^^rial re- 
sources.^ It also enables the reader to spot potential trouble areas, i.e., those 
areas with zero or negative slack time. The value of slack, associated with an 
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Expected Completioii TTimes, Latest Allowable Ootnpletion TTime and Slack 



event, will determine how critical that event is to the completion of the pro- 
gram. The smaller the slack value the more critical an event becomes. 
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^ ^ the probability of reaching the final event in the time allowed 

the rollowing formula is used: * 

2 Tb — r*E 

^ ~ \/E o^Tb 

In most program flow charts there are many possible paths from the initial 
went the tcminal one. They will vary in the total amount of slack time. 
1 lie path described as the critical path is the one with the least amount of slack 
tim^ requiring the most time to get from initial to terminal event, and causing 

delayed when any event on the path slips in time. 

The critical path in Figure 3 is indicated by a line from events 1-5-4. 

CONCLUDING COMMENTS 

The main advantage of a system such as PERT is that it forces the organi- 
Tu-ition into the development of a model which shows the activities with esti- 
mates the time necessary to accomplish the sequence of interdependent 
actions. The result is that priorities are established, staff time is allocated ra- 
tionally, budgets are estimated more precisely, bottlenecks are eliminated, and 
program objectives arc achieved more efficiently. The model also provides for 
more mpid and accurate commiu > ication between workers, divisions or agen- 
cies, since It shows the precise role of each unit in the total process. Unneccs- 
is eliminated and critical^work is more likely to be accomplished. 

_ PER r IS not a panacea. But, it it is applied to the decision-making process 
It reveals the necessary steps about which no argument is possible, and mini- 
mizes friction in determining what activities must be conducted. 
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Behavioral Criteria for Short-Term Training^ 

Leo A. Hamcrlynclc, University of Oal^ary 

In part this paper is a criticism of many evaluation studies, including* some 
of my own, but the esseiicjial point is a proposal for the solution of a major 
aspect of the key question in evaluation: Criterion Selection. iVIy data for this 
proposal derives from diverse sources. First, I have sampled a large portion 
of the liter-' the evaluation of short-term training. Next has 

been TT' tty as former training director at the University of 

Orcg< ^nining Center. The data from these sources has been 

incons ^ position as a psychologist of dct HI behavioristic views. 

An aspect uehaviorism which has been particularly active in developing 

the inconsistency is the demand for behavioral operational definitions. The 
proposed system for selection of training effectiveness criteria is directly tied 
to the process of operational definitions in a behavioristic paradigm. I suggest 
that if you hold views in conflict w^ith the behavioristic you might still 
sider the basic appeal to tlie funding public wlicre the pragmatic, utilitarian 
questioner is very evident. 

Rather than lead you through my experiences with the literature as well as 
my own research activiti^ I will fit these into the discussion of my criterion 
evaluation system. The system follows a categoru^tion based upon; 1. Prox^ 
imity to the client papulation which divides into Primary and Secondary ^ and 
2. The method of observation used in the collection of data^ which I divide 
into Direct and Indirect^ The resulting ^stem is a two-by--two paradigm. 



Dircer/Primary 


Indirect/Primary 


Direct/ Secondary 


Indirect/Secondary 



There is a hierarchy in the system which proceeds from the premise that 
short-^term training is conducted with a final goal, ^*helping people that hurt.’* 
AVhether the hurt is in terms of phantom limbs or social/occupational failure, 

^ Adapeed from a paper presented at the 75 Annual Ooavention of the American Psydto- 
logical Association, Septrember 1P67, AVashlngton, D. O., and printed with the permission of 
the author. 

o 
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It IS easj^to say, “These t’nople hurt a?,?d trainingr programs arc intended to 
train stall to alleviate these pains.’’ “Tiiis premise therefore dictates that the 
ultimate criterion of our success in training can only be found in the parameter 
Primal y" Data actual client behavior. The kind of observations^ either 
Direct or Indirect, are of obvious importance, but the resultant data is judged 
to significance only in relation to its proximity to client behaviors. 

Direct Observation-Primary Behavior 

^od literature under the heading of “evaluation” which described 
activities where the client population was directly observed. There are such 
activities, but not labeled as evaluative. The patients/c’ients of individuals 
trained in the application of behavior modification techniques are perfect 
examples of Direct/Primary category evaluption studies. The essence of such 
3 Criterion is thar a person given short-term training in behavior modification 
techniques changes the behavior of his client. The behavior of the client served 
by the trainee is the criterion. 

I would exclude self-report data from this category. It should be considered 
as primary data, but of an indirect source unless self-report behaviors arc the 
client problem. Quite obviously, many training programs are not directed 
at client behavior, but at professional activities and public relations. The 
systern will account for evaluation of such programs, but makes explicit a 
cntcrion for selection of training programs. My conservative view is that too 
often we lose sight of the final purpose for short-term training — client/patient 
welfare. 

Indlrect Observation-Primary Behavior 

”Thc criterion derived from this combination is also relatively rare in publi- 
cations and practice. Follo'w— up studies using retrievable data such as releases, 
remission rates, employment record, etc. are used to typify this approach to 
evaluation. There is danger while such data may be quite adequate for use 
with an evaluation of a. total program or system such evaluative criteria are 
too molar for short-term training- When self-report such as client attitudinal 
assessments, Q sortt, and check lists procedures are utilized, the question of 
behavioral change is too often left untested. The aforementioned use of client 
self-report or language change might be more feasible. The molarity of the 
fopner combined with the difficulty of partialing out the contribution of 
skills or attitudes acquired in short-term training from the regular training 
treatment program leaves the Indirect/Primary classification cssentiaily 
neglected. 

A. technique which I have found very promising is client self-recording- of 
bclmvior under a behavioral contract. We used a self-recording technique 
with children in a special education class under contract with their teacher 
( I^®*Tierly nek & Donley, 19^8). Although the project resulted from a graduate 
semmar, the period of ti-aining in class certainly did not exceed the number 
of hours we um in our short-term training. The resultant feedback on my 
t®®^hing c£Fectivcness was the kind that too many instructors have never en- 
countered; 1. I taught a professional some ways to help her students; 2 . she 
nsed the techniques; and 3. problem behaviors eliminated. T*he power of 

such evidence is self-evident and the data is hard. 

'rhe prerequisites are close observation of the target population and, of 
course, the content of the instruction. Expending such efforts is quickly 
justified by the feedback obtained. 
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Direct Observation-Secondary Behavior 

-A.n excellent example of evaluation, activities of this description is in trainings 
using' simulation activities. The observations are made directly with the trainee 
working on a simulation {problem. The training and evaluation are essentially 
inseparable as the criterion measure is a vital part of the simulation. 

*There is a considerable gap betw^een the behavior of a professional in a simu- 
lation problem and his client’s behavior. However, for quick feedback and 
teaching effectiveness, the simulation approach is excellent. The reality of 
high cost in time and funds, if using actual client behaviors, adds to the value 
of simulation. 

An approach of equal promise is to directly observe the trainee in his pro- 
fessional activity. Barclay p968) has evaluated the effect of an NDEA Insti- 
for School Psychologists with just such an approach. X he management 
problems are considerable, but the information is of high significance for 
planning of subsequent training institutes. 

Indirect Observation-Secondary Behavior 

This classification dominates the evaluative activities evident in the litera- 
rure. ^I^am currently engaged in just such evaluative research of our training 
activities. Such dat^i is of a secondary source, probably a questionnaire, atti- 
tude scale, content examination, or possibly several assessments all of which 
are obtained on the trainee. The data then is obtained indirectly, probably 
self-report, and about the professional in training. 

Of obvious magnitude is the unexplained variance between an attitude scale 
for a professional and the behavior of his client. The same holds true for the 
gap bc^veen information sampled in a content test and the ultimate valida- 
tion with client behavior. I have noted in my own research using a semantic 
differential technique for attitude measurement that our data hold real promise 
as a contribution to basic studies in attitudes toward exceptionality or 
ance. But, it docs not give us the evaluative information ^ 
or accelerate our program. The best evaluative data we get . 

of phone calls, letters, and foremost, visits to our trainees. The visits allow i 2 s 
to observe client behavior during programs using techniques we have taught 

the professionals. T'his is PTtTstary Data. The other feedbacks might be in- 
direct and subjective, but based upon client behaviors which is our terminal 
criterion. 

The best that attitudinal measures can ever offer to the training agency is 
predictive statement about client behaviors which would mean a saving in 
time of observation. The methodological problems in the measturement process 
alone are enough to negate this ideal point. If you contemplate the problems 
involved in obtaining validation data for trainee behavior and finally client 
behavior, the problems are of such a magnitude as to preclude such activities 
for evaluative purposes. It should remain an academic problem which social 
psychologists might attend to in pursuit of theoretical questions. 

The same criticism is true of all self-report data from the trainees. To obtain 
data for the evaluation of the instructional environment, why not simply 
count the number of complaints or ^‘glitter statements” and perhaps use an 
opinion questionnaire? But keep in mind that this is all data of very doubtful 
validity to your training goal and which can also be suspect in regard to 
reliability. 
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SUMMARY 

The evaluation of short-term training must recognize cUent behavior as a 
ctuterion.^ Any evaluation based upon data from observations of the indi- 
Aindual m traimng should be directed at specific trainee behaviors derived from 
the curnculum. Simulation training holds real promise for this purpose as well 
“ the actual traimng. Finally, use generalized evaluation with all of the 
cautions demanded by this criterion system. Don’t confuse data which is 
essentially of research nature with the kind of data you must have to evaluate 
your training program. 
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Professional Opinions Regarding Curriculum 
Content in Short-Term Training Programs in 
Mental Retardation: An Evaluation Survey^ 

Patrick J. Flanig-an, University- of Wisconsin, A/Iadison 



The present programmitig- trend in the provision of educational and habili- 
tativc services for mentally- retarded individuals and their families has necessi- 
tated a nation-wide endeavor to create and implement a variety of both short 
longf— terrn training programs for professional and para— professional per- 
sonnel working in the field of mental retardation. Since the effectiveness of 
increased^ services and ^programs is, in large part, dependent upon trained 
P*^°/cssional individuals with basic competencies in the area of mental retar- 
dation, the nature of the curriculum of these training activities is of critical 
importance. 



In reviewing the literature on the training of professional personnel in 
ment^ retardation, it becomes evident that little, if any, mention is made of 
‘specific* curiicula preparatory areas which arc deemed essential, to have had 
either didactic or practicum experience in, for working with the mentally 
retarded. Parallel to this obvious literature deficiency is the void in other 
sources concerning habilitation and special education training programs as 
they relate to specific content involved in program practices. Warren fl957) 
Hylbert 0963), Olshansky Cl^>57), Cantrell (1958), Fleischer (1964), and 
rhe Kent State Study ( 1 9<S5 ) all discuss curricula for rehabilitation counselors 
but nowhere in their descriptive statements do they contain specific informa- 
tion pertaining to the mentally retarded. Each of these studies concluded that 
the labor market and employment relations were. critically important content 
areas and^most of them mentioned that all training courses must include prac- 
tical applications and field work experiences. 

Goldman (1959), Stevens (1955), Lerner (1957), Wolinsky (1967, 1958, 



1 Paper pr^nKd at the sm entitled. *‘Scrates;ies for the Evaluation of Short-TTerm 
Txatning m Rehabilltadot^ February 1969, Rehabilitation Research and Training 
1 X 1 iVfcnul Ftecardauon* University o£ Oregfon- 
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195C>), Cain (i96*4), Olson and 1-^ah.n (19<S4), Fils and Attrwell (1966), Bles- 
sing* (1966), and Mihizzo and Blessing (1964) examined programs for edu- 
cators of the mentally retarded but made no references to rehabilitation prac- 
tices and principles in their discourse, Stevens (1963), Harrison (1963), and 
ICoIcaslca (1966) attempted to identify problems and explain programs for 
institutional stai? and camp volunteers working with the retarded. But their 
information did not contain any principles relating to rehabilitation practices. 

Two suggestions for future training programs recur throughout the afore- 
mentioned article';. These are: 1) more instruction should be given in the 

behavioral sciences, such as sociology, psychology, economics, medicine and 
education; and 2) additional involvement in concrete experiences is deemed 
absolutely necessary. 

Six authors dealt, to varying extents, with the problem of bringing rehabili- 
tation personnel and personnel w^orking with the retarded together in a training 
situation. But only one of them, Clark (1967), included in her description all 
of the professions theoretically working with the retarded although she did 
not intervene during the active training processes and attempt to discover the 
trainees’ perception of curricula needs. Younie ( 1966) and A/Ieyers (1962) 
discussed a vocatio rehabilitation curriculum for special education teachers. 
The teachers in Yo ’s study wanted more information about rehabilitation 
philosophy, terminology and patterns of local services as well as classroom 
programs and vocational guidance and counseling. The trainees at California 
Stare College, as reported by Clark (1967), represented several professional 
disciplines w^orking in the rehabilitation realm along with the retarded. Their 
primary focus w*as reviewing pc.tterns of local services and institutional visi- 
tations. Power (1962), Jaslow (1967) and Seidenfcld (1962) in their manu- 
scripts have advocated vocational rehabilitation programs for working with 
the retarded, however, little information is available in theu' sources with 
regard to the content, structure, and implementation of th^tf recommended 
programs. 

Oniy Cantrell (1958), Fleischer (1964) th " (19o^y, 

Goldman (1959), Fils and Attwell (1966^ aiiu ounic (1966) involved 
trainees, to a limited extent, in an attempt to substantiate their specific cur- 
ricula needs. Yet, only a very .small number of samplings of the trainees over 
a very short period of time were undertaken in this regard. Lerner (1957), 
Cain (1964), Warren (1957), Whitten (1957), and the two rehabilitation 
department publications (1966, 1967) all pointed out the critical importance 
of doing this. Those who did not consult active trainees were primarily in- 
terested in vocational rehabilitation counselor training programs generically 
and not necessarily programs concerned primarily with curricula having rele- 
vance for the mentally retarded. 

Thus, this project was undertaken in recognition of the paucity of existing 
descriptive literature pertaining to suggested program curricula as obtained 
from practicing professionals involved in programs and services for the 
retarded. 

If? order for any on-going educational program to effectively serve the 
population that it is designed to serve, it is necessary to periodically re-e’valuate 
the needs of curricula content and attempt to enrich programming in current 
on-going activities in an attempt to meet these needs. In addition, it is believed 
that solutions for enrichment and modification of existing programs may be 
realistically found only by consulting with those professional individuals who 
arc most cogni 2 :ant of, and cpntmually involved in, critical curricula areas. 



Curriculum EvaIuatJon/7l 



ConsequentJy, the purpose of this project was to directly sample the involved 
personnel to determine these curriculum needs and to integrate the findinirs 
iritro on— going snort-term training programs, 

THE SURVEY 

Setting 

Since 1964 the Rehabilitation Research and Training Center in Mental 
Retardation, Umversity of Wisconsin, Madison, has had as one of its major 
programs the provision of short-term training courses for professional per- 
sonnel working m the area of mental retardation. The intent of these programs 
has been to provide additional preparation in mental retardation required by 
profcssional personnel in any of the (behavioral) rehabiUtative professioris 
presently in service and to provide an orientation to, or additional preparation 
in, specific areas of mental retardation. 



Subjects 

All trainees m attendance at short-term training sessions conducted by the 
University of Wisconsin Research and Training Center in Mental Retardation 
bety^cen September 1, 1964, and October 31, 1968, voluntarily participated 
in the survey. More specifically, 3299 practicing professional individuals rep- 
resenting 1 2 professional orientations were screened in regard to their specific 
curricula suggestions which they deemed applicable to short-term training 
programs in mental retardation. All of the trainees at the time of the survey 
initiation were engaged in working with mentally retarded cIior<“? d their 
families on either a full or part-time caseload basi' able 1 ^ripi 

data on the number of • . the proicssional d cipim^o represented by 

all trainee .r m ui.r vey investigation. 



Table 1 

Summary of Professional Disciplines 
Represen red at Trainingr Institutes 
September 1. 1964 to October 31 . 1968 



VRA Counselor 7 S 3 

VRA Supervisor 173 

VRA Administrator 164 

Education 522 

Psychology 258 

Social Worlc 34Q 

O.T^ P.T., R.T.. I.T. 435 

C^encrai Administrator 197 

Physicians 47 

Mursingr 211 

Ciert^ 3 . 

Others 164 



Total 3299 



Procedure 

Ac the opemng registration of the short-term training seminars, each trainee 
' ^as requested to list ten curricula areas' that they felt would be of importanc»^ 
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to them in their professional endeavors for the following" two reasons: 1) the 
relevance of these curricula areas to situations which repeatedly arose in their 
provision of professional services to retarded clients and their families; and 
2) other behavioral characteristics of the retarded which they felt additional 
knowledge of would be beneficial to them in tlieir generic professional activi- 
ties. Approximately 30 minutes time was alloted for collection of the obtained 
stimuli descriptive data. 

The obtained survey data which was continuously collected at the begin- 
ning of all one, two and four week short-term training sessions was arbitrarily 
categorized into 15 generic content areas. Table 2 presents the generic cate- 
gorized areas which were used in tabulating the obtained descriptive data. 

Table 2 

General Categorizing Areas Used in 
Classifying Obtained Curricula Data 

I, Etiology and General Characteristics 

II. Evaluation 

III. Behavioral Techniques 

IV. Counseling 

V. Care 

VI. Recreation 

VII. Rural 

VIII. Programming and Curriculum 

IX. V ocational Rehabilitation 

X. Sheltered Workshops 

XI. Administration 

XII. Public Relations 

XIII. Research 

XIV. Therapy 

XV. Sociology and Psychology 



Elach general category was further subdivided through the addition of sug- 
gested stimuli data on the part of the trainees. Before any of the suggested 
curricula data met the criterion for inclusion in the final data tabulations it 
had to have been suggested by at least ten different trainees irrespective of 
program participation. 

Additionally^ a further categorization of the data was a breakdown of cur- 
ricula suggested by the 12 professional disciplines represented at the training 
sessions. The criteria utilized in this regard ^^further data breakdown) was 
again the recording of a curricula suggestion at least ten times, plus in the 
further breakdown analysis suggestion by members of the same professional 
discipline at least ten times. 

Analysis and Discussion 

The arbitrary classification system was developed and udlizcd after the 
data collection was undertaken for approximately one year, and observation 
of the data after a year gave indications as to the satisfactory method of classi- 
fying all of the obtained and anticipated data. Additional topical subheadings 
were added to the configuration as additional criterion content was obtained. 

A secondary categorization of the data broke down the suggested stimuli 
items by the professional discipline of the individuals suggesting the area of 
content. The criterion of an item in this breakdown, being suggested at least 
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t:en times t>y^ members of tlie same professional discipline, was imposed prior 
to the classification of a curricula item. 

The tabulated data presented in Tables 3 throug-h 17 (see pp. 78-86) sum- 
marize the 13 categorized curricula areas in the project. For clarity purposes 
they will be interpreted from two points of view; 1 ) curricula areas relating 
to basic and advanced academic information in mental retardation, and 2) prac- 
tical techniques that could easily be transferred to specific programming 
practices. 

Inspection of the majority of the tables (3—17) indicates that a majority of 
the responses obtained from this sample could be arbitrarily classified as 
falling primarily in the *‘basic information in mental retardation” category. 
This interpretation is reflected in many of the categorized titles which can 
be readily ascribed to an orientation course outline in mental retardation. A 
furtlier review of many of the subtopics under many of the headings, i.e.. 
Table 2, topic 1; Table 3, topic 8; etc., reflect num-.^rous content areas which 
are at least given some attention in the initial exposure in this respect. Addi- 
tional patterning of the sugges»:ed content areas and subpoints, i.e., evaluation, 
counseling, etc., could provide a basis for planning other training endeavors 
relating to basic information in mental retardation in a variety of training 
settings. 

•A.S a nieans of discussing and interpreting the obtained data in this project 
for clarity put poses, the following chart is presented. This chart presents an 
ititerpretive categorization of the stimuli data in this project, categorized by 
level and sequenced into a proposed sequential experimental short-term train- 
ing program in mental retardation. 

Level f, IJnit A. illustrates the beginning of the proposed sequence which a 
trainee, again from any of the rehabilitation related professions, would experi- 
ence in order to begin preparation for enriching their endeavors with the 
retarded. A basic orientation course in mental retardation would serve to pro- 
%dde a thorough introdut to mental retardation. Trainees would pursue 

a comprehensive study of the mentally retarded as a biological and socio- 
cultural phenomenon. Initial study of the definition, prevalence and etiology 
would be followed by an intensive evaluation of physical and psychological 
characteristics and associated medical, social, legal, and educational-vocational 
considerations. Various curricula resources would be available in this regard 
to permit an interested trainee to complete this requirement in a number of 
different ways,^ i.e., correspondence course, programmed manual course in 
mental retardation, etc., thus, a trainee would'not necessarily have to receive 
this training away from his home setting. 

II* Unit A would provide a basic enriching aspect through a sequence 
of visitations to specialized programs and facilities that serve the mentally 
retarded. These visitations, reflected periodically in the obtained stimuli data 
of this project, would include traditional programming practices and services 
focused around the specialized area of interest of the particular trainee.” 

Level IJI, Unit A would provide for an intern training situation with specific 
professional specialists in various programs and services dealing with mentally 
retarded cl? .'nts. Thus, a trainee would select a professional specialist who 
most closely approximates his particular interest or responsibilities in service 
area^ nr mental retardation. X he 5 >e individuals could be selected irrespective 
of program setting with the primary variable in the selection process being 
the specific type of programming that they are involved in and the trainee 
could spend a blocked period of time working with this person. 
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wm ^ rejects the active professional participation that the trainee 

will engage m upon his return to his permanent job situ^ion. 

J^vel I, Unit B would further academically orient the trainee from the 
the^^ reference of learning and the observation of learning problems as 

raiSld^Thf^ behavioral characteristics of the mentally re- 

^d^. This additional exposure would serve the following primary fun^ion 
t*®**^S» of providing adviced instruction in the general learxilng psychol- 

areas to be emphasized would^i^clude 
of°he reta'i^did.^*'^ ’ "’<>*^»'^ational, learning, and affective-personality aspe^t^ 

^ would function along the line of providing the trainee with 
rSrardatS^^^e^rr^"™^ concentrated study in the clinical fspects of m^tal 
iFo would concei'nrate primarily on the techmoues and 

problems in behavioral evaluation and diagnostic remediation treatment prob- 
le ns of mentally retarded children and adults. An in depth exposure irx pro- 
cedures for the assessment of behavioral factors would be studied. Significant 
consideration would be given to the various psychological techniques dealincr 
the numerous behavioral problems of the retarded. ^ 

fr^lareH iJrVlT ^ would involve the trainees in a related practicuin activity 
(related to the content or seminar emphasis in Level TI Unit S') which 

give prmary emphasis to the current multidisciplinary approaches to meeting 
the needs of the mentally retarded individual and hiJfa^iy. ^ 

. provide the individual the opportunity to relate 

Se Levels I, II, and III, Unit B to the specific job situatioS in which 

cnc tjrainee uiLimately ^will be retiirning*. 

Level I, Unit C would seirve to provide a varietv of functim-i<j ^ 

program could include such generS topics as the rTle of ?rderal 
agencies in the area of mental retardation, the function 

organizations concerned with mental retardation. Additionally as- 

for Re°i^e*d ^hildrelT^ service programs, the role of the National Association 
tor rCetarded Children and its counterpart at the state and local levels in meeting 
the needs of the mentally retarded would be investigated. Of primary concern 
during the conduct of this type of clinical experience would be theT^sues tW 
are evident in numerous curricula areas such as definition, prevalence etiology 
prevention reeatment, and the role of the mentally rSarded iiTour soSS^l 

of these issues in seminar fashion would permit more 

slS'il'a? ^ould fn"tS^reti?iom%S^^^^ “ 

tions undertaken by the trainees during the initial sequencing°5p iSJ^atfon" 
and would include direct participation by specialists from a Variety ofS^emal 

would add an integration Sect to the 
orgai^ation, function and contribution of mental retardation facilities Addi 

ennSh^'T^?""^" " seminar of this type would present an 

c™h interests and skills. Research needs would bedE 

cussed and the trainee would be provided with an experience in develoniig 
^ojects and, if feasible, the discussing of experiences gained dming 
rese^ch projects relating to mental retardation- ThS 
would provide trainees an opportunity to have proposed research 
progfiram demonstration projeots evaluated. 

short-term training program presents many 
P not being focused upon in moS existinJTro^ 

^ there^e a small number of prospective trained who 

would be able to obtain sufficient release time from their normal job responsi- 
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bilitics to take advantage of the total proposed programming sequence. The 
majority of the participating trainees would, at best, complete Levels I, II, HI, 
and IV, Unit A and possibly a limited exposure to portions of Unit B. Upon 
completion of Units A and B including all levels, it: is assumed that an individual 
would have developed enough knowledge and skills in the practical aspects of 
mental retardation to conduct in-service training programs in mental retarda- 
tion in their respective job settings, thereby guaranteeing an additional training 
effect. The proposed scheme would also provide a basic frame of reference 
for the completion of sequential curricula materials, i.e., instructional mono- 
graphs, audio-visual aid materials, program manuals, results of research proj- 
ects; which would have direct relevance to various sections of the tirsiining 
program. The follo^ving list represents endeavors in this regard by the Re- 
search and Training Center at AVisconsin; 1) correspondence course — Intro- 
duction to Mental Retardation (Level I, Unit ) (active); 2) An Orientation 
to Mental Retardation: A Programmed Manual (Level I, Unit A) (published); 
3) monograph — Epidemiology of “Cultural-Familial” Alental Retardation 
(Level I, Unit A) (published); 4) monograph — Comprehensive Family Re- 
l^bilitation of the Mentally Retarded (Level III, Unit B) (in preparation); 
5) monograph— Research on the Use of Behavior Modification in Ii^uencing 
the Vocational Behavior of the Mentally Retarded Client (Level II, Unit B) 
(published); 6) monograph — Handbook on Behavioral Modification Tech- 
niques for Use by Rehabilitation Personnel (Level I, Unit B) (in preparation); 
7) monograph — Changing Concepts in Mental Retardation (Level I, Unit A) 
(published); 8) monograph — Pre- Vocational Preparation for the Mentally 
Retarded (Level II, Unit B) (published); 9) monograph — Psychological In- 
struments in Use with the Mentally Ret:;rded (Level III, Unit B) (in prepa- 
ration. 

SUMMARY 

Innumerable possibilities exist in a number of different directions for en- 
riching content information. Certainly, various research endeavors can be 
easily sequenced into appropriate programming levels; this is necessary be- 
cause the basic function of short-term training programs for professional 
personnel have as their unified objective the coupling between research en- 
deavors and their practical application to habilitation through interpretive 
meclianisms in all of our on-going programs, i.e., teaching, clinical work, and 
research. 

Data derived from various professional oisciplines are depicted in Table 18 
(sec pp. 87-88). This analysis is presented for review and possible utilization 
in the planning of s(>eciflc training prog r a m s in various settings. 
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Table 3 

Etiology and General Characteristics of the Mentally Retarded 

Content Areas 



I. Basic Cau! es of Mental Retardation 

a. Genetic factors 

b. Environmental factors 

c. Oefccts related to prolonged pregnancies 

d. Effect oi delayed inter-uterine development of a child’s physical and mental ability 
2* Definition and Classification of iVlental Retardation 

a. - Intelligence and environment as related to retardation 

b. Intelligence and genetic factors as related to retardation 

3. Causes of Brain Injury 

a. Causes of other neurological impairments 

b. Learning disabilities of brain injured 

4. Developmental Aspects of Mental Retardation 

a. Language development 

b. Auditory disorders 

c. Physical development of the retarded 

d- Emotional and social disturbances in the retarded 

e. Behavioral disorders and mental retardation 
L Personality characteristics of the retarded 

Psychological problems associated with mental retardation 
h. Biochemical factors as related to retardation 

5. Historical Survey of Mental Retardation 

6. Frustration and Xolerance Levels of the Retarded 

7. Role of the Medical Profession as Related to the Causes and Prevention of Retardation 

8. Wutritional Factors as Related to Retardation 



Table 4 

Evaluat ion and Assessment of the Mentally Retarded 

Content Areas 

!• Tests 

Types of tests used with the retarded 
b. Specific tests used with the retarded 

Effectiveness of testing procedures used v/ith the retarded 

d. Methods of evaluating and assessing the retarded 

Ca Standardization procedures in developing instruments for use with the retarded 

2. Testing Procedures: When to Evaluate and Re-Evaluate the Retarded 
3* Evaluation and Diagnosis 

a. Evaluating and diagnosing the retarded 
1* potential and interests of the retarded 
2m functioning levels of the retarded 

3. social behavior of the retarded 

4 . work performance Cs’cneral) of the retarded 

P«dictjw of vocational success and placement techniques through evaluation and 
diagnostic procedures 
c* Evaluation and diagnosis of the— • 

!• autiitorily and visually impaired retarded 

CCominued on page 79> 
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Tabic 4 (Continued) 

2. multiply handicapped retarded 

3. brain damag^ed retarded 

4. learning problems in the retarded 

5. severely and profoundly retarded 

d. Identification of potential delinquents in the retar ded population 

e. Identification of reading problems in the retarded 

4. Intwpretation of the Results of Diagnosis and Evaluation by Levels of Retardation 

a. The factors, sociological and behavioral, which afiect decisions to institutionalize the 
retarded 

b. Causes of mislabeled retarded pupils (pseudo-diagnosis) 

5* System Analysis of the Diagnostic Proersss 

6. Single Disability Evaluation Centers 



Table 5 

, Specialized Behavior al Techniques Used with the Mentally Retarded 

Content A-*^cas 

1. Operant Conditioning Techniques and Training of >Xon*Professional Staff in Imple- 
menting Treatment Programs 

2. Specific Behavioral Problems and Behavior Modification Treatment Programs 

Psychiatric Medicine iti Behavioral Disorders with the Retarded 

4. Use of Positive Reinforcement: Payment or Reward with the Retarded 

5. Use of Kegadve Reinforcement with the Retarded 

6. Principles of Learning Theory as Related to Training Methods in Self-Help and Sociali- 
zation Skills with the Receded 



7. Motivational Techniques Used with the Institutionalized Retarded 

8. Adaptive Behavior Problems in Programming for the Retarded 
iVfcntal Retardation and I— nvirontn^jntal Delinquency 

10. Management and Treatment Programs of the Aggressive Retarded, with Supplementary 
Disabiliues. Epileptic 

11. j^tablislunent of Positive Attitudes by Practicing Professionals toward Treatment 
Programs in Use with the Retarded 



Table 6 

Cox\nscling 

Content Areas 

!• Ccmnseling the Mentally Retarded 

a. Counseling roles and techniques f Or the retarded 
!• re!e of the counselor 

2. differences in counseling tcchiilques according to level of retardation 

3. group counseling techniques Wd with the mentally re»irded 

4. d^elppment of communication techniques; verbal and nonverbaL with retarded 
clients 

^ wded^c^^ts**^”**^”**** vocational rehabilitation coiMiselor in counseling with re- 

{Continued on page 80) 
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Table 6 (Continued) 



6. responsibilities of t -*»ployment counselor in serving retarded clients 
b* Training counselors to increase their effectiveness in the counseling process 
c- Oclegation of responsibility for long-term follow-up counseling needs of retarded 
clients 

d* Specific counseling problems with the retarded 

1. the delinquent retarded 

2. the culturally deprived retarded 

3. the specific disability groups of retarded individuals 

4. the Ctjrrective institute parolees, labeled as mentally retarded 

5. speech problems and their influences on the counseling situation 

e. Availability of counselors to the retarded population for counseling relationships 

f. Community resources available to counselors in treatment programs 

2. Counseling Parents of the Retarded 

a. 'I'cchniques of group counseling with parents 

b. Counseling techniques in regard to^ 

1. vocational rehabilitation 

2« placement of the retarded 
3« acceptance of the child 
4. realistic expectations of the child 
5* training a child to capacity 

6. the significance of school as well as work 
7* preventive genetic measures 

c. Parental involvement and cooperation in the counseling process 

d. The retarded individual’s effects on other family members* 



Table 7 

Various Care Programs Used with the Mentally Retarded 

Content Areas 



1. Institutional Care Programs and the Mentally Retarded 

a. Institutional programs and practices oriented toward — 

1. self-care activities 

2. social adjustment 

3. preparation for independent living 

b. In-service staff training programs and practices oriented toward — 

1. nursing and attendant techniques 

2. nursing and attendant motivational levels 

3. instruction in self-care 

c. Techniques in institutional care and management of — 

the disturbed retarded 

2. the cerebral palsied retarded 

3. the severely and profcmndly retarded 
4* adolescent reurded 

5. parental contact with the institutionalized retarded 
d* Oescriprion of residential life of the mentally retarded 

e* Mental illness and mental retardation in an institution: integration or separation of 
care and treatment programs 

2. Home Care and the Mentally Retarded 

a. Dimensions of hcxnc care 

b. Care of the infant and pre-school retarded 

c. Home instructifni in sclf-care activities with the retarded 

3. Care, Treatment and Training of the Adult Retarded in Home and Community Programs 
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'Pable 8 

Recreation for the Mentally Retarded 
Content Areas 

1- L^istire Time Activities Before and After Placement 
2* Recreation and. Handicraft Programs 

3. CampPro^ams 

4. Recreation Weeds and Pro£ri*fl™s for — 
a* Sheltered V'orkshops 

b. The multi-handicapped retarded 

c. The severely and profoundly retarded 

d. The institutionalized retarded 

e. Rural adolescent retarded 

5. Qualifications of a Recreational StaiT Member 

6 . Clarification of Professional Responsibility for Leisure Time ^.ctivities with the Retarded 



Table 9 

The Retarded in Rural Areas 
Content Areas 

The Structure of Rural Programs: Grouping* Techniques 
2 * Implementation of Rural Rehabilitation Programs 
3* Rural Occupational Placement Weeds and Services 
4. The Establishment of Training Programs 

5* A^eans of Obtaining Increased Services to the Rural Retarded 
Cultural Oeprivatlon as Related to the Rural Retarded 
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Table 10 

ProgramminiT and Curriculum 
Content Areas 

1- Curricula Programming and Guides for: As Related to the Kducation of— 

a. Pre-school educable and trainable retarded 

b. Primary educable and trainable retarded 

c. Intermediate educable and trainable retarded 

d. Secondary educable and trainable retarded 

1. reading 

2. arithmetic 

3. language and vocabulary 

4. sex education 

5- driver education 
6. social skills 
/ . remedial programs 
8. work-study programs 

2. Programming and Curriculum as Related to School-A Vork Programs 

a. Elstablishment of a school- work program 

b. C'oordination of a school-work program 

c. The relation between the classroom and the work experience; classroom preparation 
and evaluation 

d» The relation between the school adminiscration and the work experience programs 

e. Specific examples of school -work programs 

f» Teaching methods for occupational ad3ustment programs 

g. Suitable materials for vocational programs 

h. Recruitment of employers for work programs 

i. Job placement of the retarded 

j. Legal implications of school-work programs 

3. Scope and Sequence in Curricula for the Retarded 

4- Suitability and Availability of Curricula Materials in Relation to Level of Retardation 

5. Teaching Techniques as Related to Level of Retardation 

6. Techniques in Developing the Self-Concept of the Retarded 

7* The Role of the Public School in Programming for the Mentally Retarded 

8. Specific Considerations in Program Planning 

a. Program planning as related to the needs of — 

1. the multiply handicapped 

2. the special learning disability group 

3. the adolescent school drop-out 

4. the culturally deprived. 

5. the emotionally disturOed 

6. the slow learners 

7. the homebound retarded 

8. the geriatric retarded 

9. the profoundly retarded 

b. Scheduling considerations 

1. apportionment of time in various academic and vocational subiect areas 

2. appropriate use of modular scheduling 

3. placement of the retarded in special and/or regular classes 

c. Programming as related to the function, structure and establishment of day care 
centers 

d. Planning a secondary program to meet the needs of Ac retarded 
c. Planning a summer program to meet Ae needs of the retarded 

f* The relation between physical facilities and program planning 

9, Techniques in the Rvaluation of Programs 
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Table 11 

Vocational Rehabilitation 



Content Areas 



!• Definition of Population Being’ Served by Vocational Rehabilitation Personnel 

2. Worlc Adjustment Programs 

a. Procedures in the establishment of various vocational rehabilitation progra^ns 
b- Pre-vocational training programs in relation to— 

1. development of social skills in the retarded 

2. development of work attitudes in the retarded 

c. Techniques of on-the-job training and supervision for the retarded 

d. Motivational techniques as related to employment for the retarded 

e. Techniques of vocational evaluation and prediction with the retarded 

f. Employment opportunities for the retarded in relation to program planning 

g. Responsibility for vocational placement of the retarded 

h. Work adjustment program — employer relationships 

1. education of the employer 

2. client information useful to the employer 

i. Research on job permanency 

3. Current Trends, T. echniques, and iVfaterials of Rehabilitative Programs 

4. RehabLitation of the Multiply Handicapped: Use of Prostheses, Etc. 

5. RehabOitation of the Severely and Profoundly Retarded: Home or Self-Employment 

6. Adult Education Programs for the Retarded 

7. Long-term Planning and Expectations of the Retarded 

8. The Relation between V ocational Rehabilitation Training and Work Programs in 
Institutions 

9. The Role of Vocational Rehabilitation Services in the Provision of Living Arrange- 
ments for the Retarded 

a. Independent living programs 

b. Halfway house programs 

c. Guest house programs 

d. Foster home programs 

10. Development and Organiz'^rioa of a comprehensive Rehabilitation Center for the 
Retarded 

11. Planning and Development of Ctwnmunity Services for Vocational Rehabilitation of 
the Retarded 

12. Quality of Services in Relation to Case Load and Administrative Duties with Retarded 
Clients 
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Table 12 

Sheltered Workshops and the Mentally Retarded 



Content Areas 



1. Development of an Extended Employment or Terminal Workshop for the Retarded 

a. Orientation and philosophy of workshops 

b. Services available in the establishment of a workshop 

c. Workshop activities and structure 

d. Management of a workshop: staff ratio with retarded clients 

e. Financial aspects of workshop programs 

f. Comparison of state and privately operated workshops 

g. Production and inventory control techniques 

h. Subcontract bidding 

1. methods of obtaining contracts 

2. vocational rehabilitation service assistance in obtaining contracts 

i. Effective marketin;? of products 

j. Relation between union and workshop employment 

k. Specific workshop clients problems 

1. management of social problems 

2. the place of the non-productive worker 

l. Geographical location of the workshop 

2. Placement Problems with the Retarded 

a. Determining eligibility and potential of workers for specific jobs 

b. Training and programming as related to placement 
1 • development of task motivation 

2. development of task persistence 

c. Type of employment as related to individual need 

d. Role of the supervisor in placement practices: supervisor-employer reelationships 

e. Vocational evaluation: its strength and weakness in relation to placement 

f . Utility of work samples with the retarded 

3. Legal Aspects of workshops Elr.iployment for the Retarded 

a. Legal responsibility of employer 

b. Wage and hour law as it affects workshops 

c. Labor laws affecting workshop programs 

d. Insurance coverage of student trainees in workshops 

4. The Function of the W^'orkshop within a State or Private Institution Serving Retarded 
Clients 

5. Methods of Facilitating the Transition from Sheltered to Competitive Employment 

6. Comparison of Sheltered Workshop Programs with Training Programs for Non- 
Employment 

7. Considerations in Terminal Employment Decisions with the Retarded 

8. Requisites for Staff and Staff Training Programs in Mental Retardation 

9. Tours of Sheltered Workshops 

10. Case History Studies of Successful EImployment Programs with the Retarded 
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Table 13 

A dministxation 

Content Areas 

!• Muld-Disciplinary Approach in Provision of Services to the Retarded 
a* Coordination of services in communides 

b. Role of specific personnel in servicing* the mentally retarded 
!♦ vocational rehabilitation administrators and counselors 

2« teachers 

3. sheltered workshop personnel 
4« guidance counselors 
5m medical personnel 
6m special educadon personnel 
7. therapists 
Sm nurses 

9. non-professional aides and volunteers 

c. Communicadon between professional disciplines 

2m Staff Training 

a. Definition of personnel qualificadons for work with the retarded 

b. Recruitment of skiRiid personnel 

c. Types of in-service training programs 

d. Academic courses in staff training programs 
€• A/fethods of re-evaluadng teaching techniques 

f* A^Iethods of implementing new teaching techniques 
g. Staff libraries in relation to personnel training 

3. Administrative Practices and Techniques 

a. Cooperative programs involving several government agencies 

b. Role of the state department of public instruction in the development of a school 
progrram 

c. Supervision for professional and non-professional personnel 

d. Fostering client-centered staff communication 

e. Practices in acquiring materials and funds 

f* Practices in fostering creativity and resourcefulness 

4. Finances 

a. Federal and state funding possibilities for the retarded 
b- Grant application procedures in mental retardation 

c. Prep^tion of budgets for special education with an emphasis on less record keeping 
for school and vocational rehabilitation personnel 

d. Firanci^ aid for the unemployed retarded living at home 

c. Production costs for competitive and sheltered employment 

f. Information concerning fellowships for advanced degrees in mental retardation 



Table 14 

Public Relations 

Content Areas 

Ic Methods of Encouraging Community Knowledge and Acceptance of— 
a. School-work programs 
hm Sheltered workshops 

Cm ocational rehabilitation tramuig programs 
d. Employment of retarded individuals 

2. Methods of Eliciting Community Support of, and Intere* in. Programs for the Retarded 

3. Use of Existing Community Resources 

4. Utility Group Work Activities in the Community 
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Table 15 
Research 



Content Areas 

1. Review of Current Programmatic and Longitudinal Research in Mental Retardadon 

2. Specific Research Areas of Interest 

a. Research on the mildly and moderately retarded 

b. Mental retardation and cultural-familial factors: core area research 

c. Mental retardadon and genetic factors: prevention oriented research 

d. Mental retardadon and eugenics: pros and cons 

e. Nutritional factors and the retarded child 

f« Alental retardation and vocadonal rehabilitadon research 
g- Mental retardadon and social work research 

h. Mental retardadon and alcoholism 

i. Mental retardadon and crime, welfare programs, and school drop-out rates 
j- Review of current University of Wisconsin research programs 

3* Integradon of Research into Action 

4. Medical Research as Related to the Incidence of Mental Retardadon 
Techniques of Data Gathering and Processing 



Table 16 
Therapy 

Content Areas 

1. Speech Therapy and the Mentally Retarded 

2. Physical Therapy and the Mentally Retarded 

3. Occupadonal Therapy and the Mentally Retarded 

4. Psychotherapy and the Emodonally Disturbed Mentally Retarded 

5. Adjunedve Therapies in Hospital Programs 

6. Integration and Coordination of Therapeutic Services 

7. Activities as Therapy 

8. Mate^dals for Therapists 

9. Group Therapy Techniques 

10. Electroshock Terapy 

11- Orthopedic Problems and Therapy 

12. Convulsive and Epileptic Problems and Therapy 

13. Pharmacology and Therapeutics 



Table 17 

Sociology and Psychology 
•“ Content Areas 

1. Sociological Concepts Related to Mental Retardation 

2. Psychological Aspects of Mental Retardation (Needs and Weaknesses in Adult Re- 
tarded) 

3. Psychologist’s Role in the Rehabilitation Progranuning for the Mentally Retarded 

4. Psychiatric Problems Associated with Mental Retardation 

m: 
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Table 18 



The Conte nt Interests of the Single Disciplines 

Vocational Rehabilitation Vocational Rehabilitation Educators 

Administrators Counselors 

<Content) (Content) (Content) 



!• V^ocational Rehabilitation 

2. Administration 
3 • Sheltere d W or I^shop 

4. Counseling 

5. Programming & 
Curriculum 

6 . Public Relations 

7. Rvaluadon 

8. Etiology 

9. Recreation 

10. Research 

11. Behavioral Xechniques 

12. Care 

13. Therapy 

14. Rural 

15. Sociology & Psychology 



1 . Vocational Rehabilitation 

2. Counseling 

3. Evaluation 

4. Administration 

5. Sheltered Workshop 

6. Etiology 
7- Programming & 

Curriculum 

8. Public Relations 

9. Behavioral Techniques 

10. Rtiral 

11. Care 

12. Research 

13. Recreation 

14. Sociology & Psychology 



(continued) 



!• Programming & 
Curriculum 

2. Vocational Rehabilitation 

3. Administration 

4. Sheltered Workshop 

5. Etiology 

6. Evaluation 

7. Counseling 

8. Public Relations 

9. Rural 

10. Behavioral Techniques 
21. Research 

12. Recreation 

13. Care 

14. Therapy 

15. Sociology & Psychology 



Table 18 (continued) 

The Content Interests of the Single Disciplines (continued) 



P^chologists Clergy Medical Personnel 

(Content) (Content) (Content) 



Therapists 

(Content) 



1. Evaluation 

2. Vocational 
Rehabilitation 

3. Programming & 
Curriculum 

4. Behavioral 
Techniques 

5. Sheltered 
IVorkshop 

6. Counseling 

7. Etiology 



1. Programming & 
Curriculum 

2. Care 

3. Recreation 

4. Etiology 



1. Etiology 

2. Vocational 
Rehabilitation 

3. Therapy 

4. Administration 

5. Care 

6. Behavioral 
Techniques 

7. Research 



1. Therapy 

2. Etiology 

3. Administration 

4. Recreation 

5. Vocational 
Rehabilitation 

6. Counseling 

7- Behavioral 



8. Care 

9. Public Relations 

10. Administration 

11. Therapy 

12. Research 



8. Sociology & 
Psychology 



Techniques 

8. Care 

9. Sheltered 
Workshop 

10. Sociology & 
Psychology 



(continued) 
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Table 18 (continued) 

The Content Interests of the Single Disciplines (concluded) 



Business Personnel 
(Omtenc) 


Sheltered Worksht^ 
Personnel 
(Content) 


Nurses 

(Content) 


Social Workers 
(Content) 


1. Vocational 


1. Sheltered 


1. Care 


1. 


Vocational 


Rehabilitation 


Workshop 






Rchabiliattion 


2. Sheltered 


2 . Vocational 


2. Programming & 


2. 


Counseling 


Workshop 


Rehabilitation 


Curriculum 






3. Programming & 


3. Counseling 


3. Behavioral 


3. 


Programming & 


Curriculum 




Techniques 




Curriculum 


4. Etiology 


4. Programming & 


4. Sociology & 


4. 


Etiology 


• 


Curriculum 


Psychology 






5. Administration 


5 . Care 


5. Vocational 


5. 


Sheltered 






Rehabilitation 




Workshop 


6 . Evaluation 


6. Rural 


6 . Administration 


6 . 


Evaluation 


7. Care 


7. Etiology 


7. Counseling 


7. 


Recreadon 


8. Counseling 


8. Recreation 


8. Etiology 


8. 


Public Reladons 


9 . Therapy 


9. Research 


9. Therapy 


9. 


Research 


10. Public Relations 


10. Public Relations 


10. Research 


10. 


Administradon 




11. Administration 




11. 


Therapy 








12. Care 








13. 


Behavioral 










Technique;. 
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Alkin, M. C. Evaluating the cost-cflFectiveness of instructional programs. Sym- 
posium presented at the Center for the Study of Evaluation, Report Nof 25. 
University of Califorma at Los Angeles, May 1969. 

Describes the application of a cost-effectiveness model to the evaluation 
of educational programs and systems. Alkin delineates the difference be- 
tween co^-benefit analysis and cost-effectiveness c^'aluation. A model 
for cost-^ffcctiven^ evaluation in education is outlined, and the possible 
uses of this model in diverse evaluation situations are indicated. 



Altdn, M. C. Towards an evaluation model: A systems approach. Center for 

me Smdy of Evaluation. Report No. 43, University of California at I.. 0 S 
Angeles, February 1968. 

Defines, discusses, and amplifies a definition cf evaluation. 

Alkin, M. C, Bentzen, M. M., & Grigsby III, J. E. CSE simulated evaluation 
exercise: Instructi^ j^ide. Center for the Study of Evaluation, Report No. 
49, Umversity of California at Los Angeles, in press. 

Describes an exercise which develops evaluators* flexibility as they respond 
to the problems they find in ai^l field conditions. Directions are mven 
for the construction and modification of evaluation designs. There is a 
disciusion of the ^ecific objectives of the exercise and scoring procedures 
for determining how successfully each objective was met. 

Alkin. M. C.. Bentzen, M M., & Grigsby HI, J. E. CSE simulated evaluation 

Center for the Study of Evaluation, Report 
No. 50, Umversity of California at Los Angeles, in press. 

Prints necessary informational material for the conduct of an actual 
evaluation* 



An^e\vs, K. Is ma^gement training effective? Part I. Harvard Business 
Revtrw, January-February, 1957, 85-94. 



Discusses the efficacy of university courses and in-service courses for 
employees of Humble Oil and Refining Company and Westinghouse. 
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These programs were evaluated in terms of the participants' reactions, 
faculty reactions, and company reactions. * 

Andrews, K. Is management training effective? Part II. Harvard Business 
Review, March-April, 1957, 63-72. 

Discusses measurement, objectives, and policy. An attempt is made to 
derive conclusions from objective criteria in order to adequately evaluate 
problems of measurement. 

Angel!, D. J., Shearer, J. W., & Berliner, D. C. Stttdy of training performance 
evaluation techniques. Technical Report: NAVTRADEVCEN 1449-1. 
Port Washington, New York: U.S. Naval Training Device Center, 1964. 
Summarizes current N^avy proficiency-evaluation methods. Performance 
evaluation in training situations which involve simulators and other com- 
plex equipment is discussed. The three variables which are important in 
the development of “a ^stem of performance evaluation” are described, 
and an “automatic training/evaluation ^stem” is explained. * 

Baker, R. L. Curriculum evaluation. Review of Educational Research. 1969 
53^ (3), 339-358. 

Reviews curriculum research and evaluation. Author points to the need 
for the differentiation of curriculum into definable and rcsearchable sub- 
ject matter. There is an emerging technology for the analysis and the 
evaluation of curriculum development which vvill clai ify the relationship 
between curriculum research and evaluation activities. 

Baxter, B., Taffc, A. A., 8c Hughest, J. F. A training evaluation study. Person- 
nel Psychology, 1953,403-417. 

Coi^P^res the effects of two conference training programs for a simple 
posiaon,^ debit imnarance agent. One group was provided conference 
training in centralized schools wliile the other group was similarly trained 
under local supervuion. Comparisons were made with regard to produc- 
tion, job satisfaction, termination rate, life insurance knowledge and 
supervisors’ ratings. An analysis of results, conclusions, and recommenda- 
tions is presented. 

Belman, H. S., & Remmers, H. H. Evaluating the results of training. Journal 
of the American Society of Training Directors, 1958, 12 (5), 28-32. 

Li^ eight areas which can be evaluated. Reviews briefly some of the basic 
principles that ought to be observed in evaluation. The authors concluded 
that procedures and techniques of evaluation must be dealt with if evalu- 
ations are to be meaningful. 

1 ^** *^*®**» ^ D* C. Evaluation techniques for management 

programs. Journal of the American Society of Trainins Di- 
rectors, 1959 , 13 (10). 13-27. ^ 

Discuss^ considerations which relate to evaluation. The evaluation proc- 
consists of definitic;i of goals, determination of training needs, con- 
struction of the program, and evaluation of methods. 

Bloom, B. S. (Ed.) Taxonomy of educational objectives: The classification of 
educational goals. Handbook /.* Cognitive Domain. New York; Longmans, 
Green and Co., 1956. ^ 

Oasifitt educational goals in the cognitive domain. The major headings 
of the classification are knowledge, comprehension, appUcation, analyf.t^ 
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evaluation. Sample test items for each category are pro- 

Bl^enfeld, W. S. Attitude change as a criterion in training. Training and 
Development Journal, 1966, 20 <9), 26^ 32. ana 

Reviews literature concerned with attitude change as a criterion in the 
evaluation of traimng. ^phasis was placed on an assessment of the 
adequacy of experunental design and the degree to which safe inferences 
can be made from results. 

results of supervisory training. Personnel, January 

Demonstrates the effectiveness of a training program by evaluating the 
resjmnses of superiors and subordinates of the trainees who participated 
m the rapervisory training workshops. The reliability and validity of 
the testing instrument are also discussed. 

5*5 (^) validity of an evaluation program. Personnel, 

Judgments of a ttainee’s performance after training usually involve opin- 
lom of his superiors and subordinates. Such judgments are used to measure 
a trainmg program’s effectiveness. This article presents the findings of a 
Study designeu to test the validity of this criterion. 

evaluating personnel training. Public Personnel 

Statw the purpose and efficacy of measurement and evaluation of training 
and describes the distinction between them. There is a need for achieved 
ment measimes which avoid common statistical errors. Measurement cri- 
teria mu^ be selected carefully, and procedure must include the before 

level of performance and standards of performance toward 
which traimng is directed. 

L. Evaluating training programs-the state 
of the art. Trammg and Development Journal, May 1968, 2-9. 

Rewiews research on current techniques being used by business, industry, 

cvaiuaoon of their training programs. Evaluations 
i”i reactions of the trainees to the programs, (2) problems 

fn^oK^K h “ trammg setting, (3) evaluation of possible change 

misbehavior as a result of exposure to a program, and (4) results to be 

Qos, M. Evaluation of mental health rvorkshops in Kentucky. Journal of Edu- 
Remits measured attitude changes in teachers who participated in a 
thi- fourth-month workshop on mental health by means of 

ar Inventory taken at the bepnning and 

at the end of the workshop and repeated nine months later. Positive atti- 

retained over the nine months, and proportionately 
g r awtude change was demonstrated among younger and less ex- 

SSTd^^'Tge 'he longer w^rfahof eh“wen,o« 

CmIc, p. L. P£RT IProgram Evaluation and Review Teehni^l; AbbUce- 
ttons m educatum. Washington, D. C: Office of Education,^o!ee/states 
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Department of Health, Kducation, and Welfare, 1965. 

The basic concepts, techniques, and methods of PERX are described, and 
the author postulates potcndal uses of PERT in a variety of educational 
and developmental activities. 

Cox, J. A. Application of a method of evaluating training. Journal of Applied 
Psychology, 1964, 48 (2), 84-87. 

Describes the Ward Edwards’ formulation of value of training used to 
process data in this experimental study. Estimates of proficiency level 
attained, worth of a trained man in dollars, and training costs in dollars 
are presented. An evaluation of the technique and results of the study 
are reported. 

Crawford, M. P. Concepts of training. In R. M. Gagne (Ed.) Psychological 
principles in system development. New York: Holt, Rinehart and Winston, 
1962, 301-342. 

Presents an overview of seven major steps in a training development 
project. These are described as: analysis of the system, analysis of the 
particular job, specification of knowledges and skills, determination of 
training objectives, construction of the training program, development 
of measures of job proficiency, and evaluation of the training program. 

Cronbach, L. f. Course improvement through evaluation. Teachers College 
Record, 19o3, 64 (8), 672-683. 

Describes two functions of evaluation— course improvement and a means 
of judging individuals. The purpose of the foxmer is stressed and several 
approaches are presented. 

Cronbach, L. J. Evaluation for course improvement. In R. W. Heath (Ed.) 
Neve curricula. New York: Harper and Row, 1964, 231-248. 

Defines evaluation as the collection and use of information for the purpose 
of making decisions about coturse improvement and about individuals. 
Suggestions are given for the use of a process study for evaluation of the 
classroom and for course evaltiation. 

Daugherty, W. D. Statistics: A method of influencing attitudes. Journal of 
the American Society of Training Directors, May 1960, 36-39. 

The author asserts that while improved instruction is significant, the 
evaluation of training and analysis of results is more important. It is recom- 
mended that those who are skeptical about evaluation be persuaded by 
the proper demonstration of the use of statistics. A variety of techniques 
are presented. 

Dressel, P. L.. Evaluation procedures for general education objectives. Edu- 
cational Record, 1950, 31, 97-122. 

Discusses ways of integrating evaluation and instruction. The author dis- 
cusses 20 questions in which testing and instruction are interrelated. 

Dressel, P. L>., & May hew, L. B. Evaluation as an aid to instruction. In S. J. 
French (Ed.) Accent on teaching. New York: Harper and Brothers, 1954, 
297-320. 

The main foci of evaluation are program effectiveness, educational pro- 
gram improvement, and student motivation. T here is discussion of the 
major purposes of evaluation, its relevance in general education, applica- 
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tion of trechniques of evaluation, in teaching'^ and guides fox the use of 
concepts of evaluation. 

Duel, H. J. Hidden indicators of training effectiveness. Journal of the Ameri- 
can Society of Training Directors^ March 1959, 3-5. 

Recommends indirect measures of training effectiveness such as changes 
in tiurnover, output, absenteeism, grievances, rejects, etc. as vital sources 
information for a comprehensive evaluation. A description is given of 
or Training Needs Rating Form that was developed for part of the process 
of data collection. 



Fredneksen, N. Proficiency tests for training evaluation. In R. G. Glaser (Ed.) 
^ychological research in training and education. Pittsburgh: University of 
Pittsburgh Press, 1962, 323-346. 



rhe a^essment of training, including the following types of 
training evaluation measures: (1) soliciting opinions, (2) administering 
amtude scales, (3) measuring knowledge, (4) eliciting related behavior, 
(5) eliciting what-would~I-do behavior, (6) eliciting lifelike behavior, 
and (7) observing real-life behavior. 



Fryer, D. H. Training, and its evaluation. Personnel Psychology , 1951, 29-37. 
Discusses methods of identifying behavior necessary for job accomplish- 
ment. Techniques c., interviewing and observing are presented. An analy- 
sis of probable attitude-knowledge-skill patterns of the job is discussed. 

Gilbert, J., Campbell, H. G., & Oliver, A. E. A.n evaluation of interdepart- 
mental training with objective tests. Journal of the American Society of 

Training Directors, May 1963,46-5^. 

Discutes and evaluates tests developed and used in primary interdepart- 
mental ttaining courses. The evaluation covers a description of the student 
population, item analysis, and the results of testing group meetings. 

Gl^er, R., & Klaus,^ D. J. Proficiency measurement; Assessing human per- 
formance. In R. M. Gagne (Ed.) Psychological principles in synem devel- 
opment. New York: Holt, Rinehart and Winston, 1962 , 419.474. 

Describes the role of proficiency measurement in the context of system 
development. Specific treatment is given to: (1) criterion vs. norm- 
referenced measures, (2) uses of proficiency measurement, (3) definition 
of the behavior to be measured, ( 4 ) sampling and the relative importance 
of performance components, (5) precision and relevance in proficiency 
measurement, (6) eliciting behavior for measiurement, and (7) some 
applications of proficiency measurement. 

Goodacre, D. The experimental evaluation of management training. Person- 
nel, May 1957, 534-538. 

Presents the B. F. Goodrich Company evaluation of a management train- 
ing program. Attitude scales were used to evaluate (1) attitudes toward 
^e company, (2) amtudes towind employees, (3) job satisfaction, and 
seli--conndence m dealing^ with a variety of situations^ Achievement 
tests were used to evaluate knowledge gained in the program. Ratings 
by immediate superiors were used to evaluate job performance. 

Harmon, F. L., & Glickman, A. S. Managerial training: Reinforcement through 
evaluation. Public Personnel Review, 1965, 26 (4), 194-198. 

The evidence of the efiFects of training on job performance is discussed, 
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and a method of evaluation is demonstrated on a training^ course. The 
evaluation process is employed to encourage positive reinforcement of 
what has been learned. 

How to get better trainee performance apprais. 's through the “critical inci- 
dent’ technique (training methods 10). The Training Workshop. Water- 
ford, Connecticut: Bureau of Business Practice, 1967. 

The lack of sufficient time to observe trainees is a frequent problem to 
supervisors. Noticing and recording critical incident performance by the 
trainees is one solution to the problem. Examples are given to illustrate 
both satisfactory and unsatisfactory critical incidents and an example of 
the form for recording them is presented. How to handle various reactions 
of trainees to their ratings on job performance is discussed. 

Hull, T. F. What should we evaluate? Journal of the American Society of 
Training Directors, January 1959, 38-39. 

Cites criteria for evaluation. On-the-jdb application of the training pro- 
gram is the goal, and a discussion of the application of evaluation is pre- 
sented along with some alternatives. 

Junker, E. S. Hiring, training, and evaluation of instructors. Journal of the 
American Society of Training Directors, April 1964, 23-30. 

Presents and discusses important criteria relating to the hiring and train- 
ing of instructors. A technique for the evaluation of instructors with the 
use of a rating scale is recommended, and an interpretation of evaluation 
results and the development of a salary schedule are presented. 

Kirkpatrick, E>. L. Evaluation of training. In R. L. Craig and L. R. Bittel (E^s.) 
Training and development handbook. New York: McGraw-FIill, 1967, 
87-112. iJoumal of American Society for Training and Development, No- 
vember, December 1959, January, February 1960, 4 parts). 

Discusses four steps considered to be critical in the evaluation process. 
The steps include: (1) reaction (how well the trainees like a particular 
training program), (2) learning (what principles, facts, and techniques 
were understood and absorbed by the conferees), (3) behavior (changes 
in performance which can be attributed to conference activity), and 
(4) results (reduction of costs, reduction of turnover and absenteeism, 
reduction of grievances, increase in quality and quantity of production, 
and improved morale). Emphasis is placed on the techniques which 
training directors can use to evaluate their own programs. 

Kirkpatrick, D. L. How to start an objective evaluation of your training pro- 
gram. Journal of the American Society of Training Directors, 1956, 10 
(3), 18-22. 

Stai.cs how an objective evaluation of a training program begins. The 
question, “Has the trainee learned the desired facts and principles?” is 
answered by four evaluative processes: use of a paper and pencil test, 
before and after tests of the trainee, comparison of each trainee’s pre-test 
and post-test scores, and analysis of item change from pre-test to post-test. 
Explanation of the statistical findings is gpven. 

Korb, D. How to measure the results of supervisor training. Personnel, March 
1956, 378-391. 

Reviews information about research methods, field experiments and sur- 
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vcy techniques for developing and measuring supervisory training. Two 
simple plans are presented — before training and after training. 

Korb, L. D. How to measure results. In XJ. S. Civil Service Commission Per- 
sonnel Aiethods Series No. 4 Training the supervisor: A. g^ide on hoiv to 
set up and conduct a supervisory training program. Washington, D. C.: U. S. 
Government Printing Office, 1956. 

Describes the field study which is the most likely approach to be used for 
scientific evaluation of training. Its components are enumerated, and in- 
structions for the use of the evaluator and charts on supervisory training 
evaluator and sample plans for their application before and after training 
are presented. Objective recording, the use of a control group, complete- 
ness of information, and follow-up are among the procedures stressed. 

Kunze, K. R. Forced choice evaluation of a training program. Journal of the 
American Society of Training Directors ^ August 1958, 27-32. 

Describes the use of a binary measuring technique for training evaluation. 
A justification of this approach along with a variety of illustration scales 
Ls presented. 



Lawske, C. H., Bolda, R. A., & Brume, R. L. Studies in management training 
evaluation. Journal of Applied Psychology, 1959, 43 (5), 287-292. 

Reports five studies conducted to evaluate the effects of single and re- 
peated exposures to the skit-completion method of role playing. Evalu- 
ation criteria consisted of scaled responses to a standard human relations 
training case in two dunensions: sensitivity and employee-orientation. 
Criterion responses were obtained before and after role playing in four 
•subject groups and after the training in the fifth group. Various role 
playing treatment conditions and role assignments were investigated. 

Lerda, L. W., & Cross, L. W. Performance oriented training— results, measmre- 
ment and follow-up. Journal of the American Scdety for Training Direc- 
tors, 1962,16 (8), 12-21. 

Results are measurable when a training need or problem has been ex- 
plicitly delineated and when there is proper definition of the program 
objectives. Areas for measuring are learning and application. Included 
in the discussion are principles for training progress evaluation, informa- 
tion about evaluating results, and a scheme for measurement of results of 
orgatiized training. 



Eippitt, G. Lr., AilcCune, S. D., & Church, L. D. Attitudes of training directors 
toward the application of research to training programs. I^eadership Re- 
sources, Inc,, Washington, D. C., or Training Directors Journal, March 
1964. 

Assess^ the attitudes of training directors on a variety of issues which 
are critical to a comprehensive training program. In this evaluation are 
opinions concerning the purposes of training, methods used in evaluation, 
uses of evaluation material, preferences of evaluation methods, and ob- 
stacles to training, research, and research needs. 



Lundberg, C. C. Attitude change and management training. Personnel Ad- 
ministration, 1962,25 (3), 35-43. 



FRir 



Maintains that the concept of attitude as currently conceived by those 
responsible for human relations training for managers is not compatible 
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training objectives. The author shows how an 
attitude actually may be a barrier to effective management action. 

S" ^ and evaluation. Training for develop-^ 

Illinois: Richard D. Irwin, Inc. & The Dorsey Pre£, 

Discusses how post-training contracts and services help both partici- 
F^nts and program after a training program in the transfer of learning 
to the work situation. Reliable criteria and evaluation procedures are 
necess^ in the determination of what post-training contracts are indi- 
VI ■ consatute the major purpose of the training and should have 

solid criteria and evaluation procedures. Measurement of end-of-course 

i“d ffft<Svene^ “ evaluation of the training process 

evaluation on training programs. 

Describes the many ways of evaluating the results of training, and recom- 
mends recognition that some methods are better than others. Classifies 
evaluation procedures and criteria in descending order of acceptability. 

^**345””^" * i^valuation of training. Personnel Journaly i960, 3S (9), 344- 

Evaluates a management training program developed by K. Korman of 
the University of Minnesota and T. H. Jerdee of the University of North 
Carolina under the direction of T. A. Mahoney of the Uifive.sity of 
M nnesota. Course objectives are outlined. Improvement is measured by 
a knowledge test, a case problems test, and an aptitude scale. 

Mahoney, T. Jerdee, T. H., & Korman, A. An experimental evaluation of 
management development. Personnel Psychology y 1960, 13 (1), 81-98. 

Describes an expermental approach to the evaluation p. ocess for measur- 
ing a large industrial organization’s program. The multiple objectives of 
the training program are recounted. Training included case analysis, 
group discussion, ^pplemental reading, and lectures. The evaluation of 
the co^se is intended to show whether course objectives were obtained 
^d whether the participating manager’s superior” helped in the process. 
The course does not meet all the objectives. The “case approach and the 
bos^involved approach” are not completely successful and re-open the 
question of the value of using an analytical approach in the training. 

oI" training. Training in business 

ana industry. New York; Wiley, 1961. 

Discuses the purpose and procedures for training evaluation. Classifica- 
tion of four typM of evaluation measures and their respective features 
are delineated. All evaluation measures should be relevant, reliable, free 
xrom bias^ and practical. 

P«*Wem in the evaluation of instruction: Assessing 
Symposium presented at the Centu 
Los^gfStMay W9 ’ University of C^Ufornia. at 

effective reaction as important cri.erion 
v^iables which should be taken into account in evaluation of instruction. 
Messick stressed these variables because of their relevance to opiniom 
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about the variab.'lity of human performance and the place of values in 
educational research. 

The evaluation of education and training. Self and service en~ 
rtchrnent through federal training, an annex to the report of the presidential 
task force on career advancement. U. S. Civil Service Commission, Wash- 
ington, D. C.: U. S. Government Printing Office, 1967, 430-443. 

States that specific program objectives are essential to the evaluation 
process. Xhe technical contribution of evaluation is the development of 
tools for the measurement of program objectives. Goals of evaluation and 
problems in measurement are discussed. 

^^966^20 relations training. Training and Development Journal, 

Reviews research on four categories for determining the effects of on- 
the-job human relations training. Evaluations of on-the-job human rela- 
tions training are difficult because of a poverty of valid and reliable 
measure.', because organizations are reluctant to cooperate in such en- 
deavors, and because skills in supervision are slow to change. 

Moon, C. & Hariton, T. Evaluating an appraisal and feedback training 
progrram. Personnel, 1958- 55, 36-21, 

States that many companies believe that evaluating the effectiveness of 
a training program demands elaborate, expensive procedures. This article 
reviews the results of a study which demonstrated the way meaningful 
results can be obtained from relatively simple measuring methods. 

Training evaluation — it used to be so simple. Journal of the 
American Society of Training Directors, April 1964, 45-50. 

Recommends a^ team approach to the evaluation of training. Inspectors, 
supervisors, tr^nees, and ixaining officers need to evaluate in accordance 
to^ specifi ed criteria. The importance of effective management in evalu- 
auon is stressed. The author suggests that only two end results be 

me^ured: quality and quantity of work on the job, and growth in the 
individual. 

Morgan, B., Holmes, G. E., & Bundy, C. E. Evaluation in adult education. 
Methods tn adult education. DanviUe, Illinois: The Interstate Printers and 
Publishers, 1960. 

Classifies types of evaluation in terms of degrees of formality and pre- 
cision. Informal evaluation includes observation and informal tests; semi- 
lormal includes reliable and valid tests and surveys; formal includes re- 
research which involves knowledge of statistical procedures and special 
traimng. Principles of evaluation are fully discussed as are the seven steps 
in evaluation. A paradigm for evaluation and a short form for the evalu- 
ation of meetings are presented. 

Nomian, J. H. Dollars and cents evaluation of a training program. Journal of 
the American Society of Training Directors, October 1959, 32-36. 

Presenmtion of a formula that yields an acceptable monetary appraisal of 
e’’'?*luation efforts. 

Qd — mac yG. S.^ A systems approach to training. JottrmA of the American So- 
^ Training Directors, October 1965, 11-19. 

an argument for a systems approach to training which concerns 
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inputs, processes, and outputs. An explanation is given of eight kinds of 
training ^sterns along with a guide for their adoption in a variety of 
training situations. 

Pace, C. R. Evaluation perspectives. Center for the Study of Evaluation, Re- 
port No. 8, University of California at Los Angeles, December 1968. 

Discusses evaluation development and the role of the evaluator in that 
development. The author, examines the new emphasis in evaluation and 
the relevance of different evaluation models to other units which, in turn, 
were subjects for evaluation. 

Parnicky, J. J. The evaluation of institutes: A g^ide for measuring their impact 
on social vuork participants. New York; National Association, of Social 
Workers, 1966. 

A guide for the aid of social work institute planning committees prepared 
as an auxiliary to the NASW Manual for Educational Directors and 
Seminar Leaders. Primary emphasis is on written techniques, with some 
attention given to structures and open-ended oral and observational tech- 
niques. 



Pearson, J. B. (Ed.) The analysis of short-term seminars in psychiatry for 
non-psychiatric physicians: A progress report for the years 1963-1966. 
Boulder, Ccloradc: ^Vestern Interstate Commission for Higher Education, 
1966. 



Questionnaires from eight seminars in 1963-1964 are compared with ten 
seminars in 1964-1965, then analyzed for their implications for evaluation. 
Two questions are fundamental in all program evaluation: What is ac- 
complished by the program? and How? 



Randall, L. K. Evaluation: A training dilemma. Journal of the American Soci- 
ety of Training Directors y 1960, 14 (5), 29-35. 

Presents various points of view concerning the evaluation of formal train- 
ing. These views include: (1) either impossible or unnecessary, (2) the 
only answer, and (3) important despite unacceptable techniques. The 
author suggests an alternative, the “refinement” type of evaluation, which 
consists of three elements: verifying basic training assumptions, estab- 
lishing satisfactory criteria, and implementing a meaningful evaluation 
design. 




Renuners, H. lA. Honv to evaluate training programs in business and industry. 
Lafayette, Indiana: Purdue University, n.d. 

Presents the purpose of evaluation. The objectives of evaluation, test 
construction, and the determination of a test's efficacy are themes de- 
veloped in four booklets from the Purdue University" Press. ■' 

i 

Rose, H. C. Evaluation of the training program. The development and super- ( 

vision of training programs. Chicago: American Technical Slociety, 1964. 
Explores three aspects of the evaluation process: the plan, the process, 
and the product. A model of a trainee evaluation questionnaire is pre- 
sented, as is an explanation of the controlled research method of evaluation 
along with two reports of the method. ! 

Rose, H . C. A plan for tiaining evaluation. Training and Development Journal, I 

1968,^2(5),38-51. 

Reviews eight elements of training and 13 techniques of obtaining infor- f 
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tnation for evaluation with a statement of their respective advantages and 
disadvantages. The author suggests that the results of various evaluative 
actions be combined to obtain the best possible total evaluation at the 
most reasonable costs. 

Sammons, R, F. Evaluating a workshop in training skills. Journal of the Amer- 
ican Society of Training Directors, July 1965, 27-33. 

A description of an evaluation procedure using questionnaires. 

Scheerenberger, R. C. An introduction to PERT and its application to educa- 
tional programs. The Winnower, 1966, 2 (4), 12-22. 

Describes the important features of Program Evaluation and Review 
Techmque (PERT). It is a structure for the work organization which 
haa generalizability to the management of programs. Its adaptability- to 
provwions of control over such variables as time, cost, and performance 
is pointed out. 

Schlesinger, L. Evaluatmg the content of multiple-skill traininir nrourams 
Personnel Administration, 1958, 21, 20-27. 

Describes three methods of appraising the relevance of training pro- 
grams to skills and knowledge required for the performance of tasks. The 
criterion used was a rating scale* TThe second was a comparison of 
the time di^ribution of training topics and relative training emphasis 
with the skills and knowledge required on the job as inferred from job 
description data. The third criterion was selecting a small number of 
critically important job activities and comparii.^ these activities with the 
actual content in the courses. 

Schultz, D. G., & Siegel, A. I. Post-training performance tnlterion develop- 
ment and application: A selective review of methods for measuring indivtd- 
iMl differences in on-the-job performance. Wayne, Pennsylvania: Applied 
Psychological Services, 1961. 

Reviews currently employed methods for cn-the-job evaluation. Two 
necessities are: greater cohesion and congruence in assessing the field, 
and a cogent evaluation of measuring techniques. 

Schultz, D. G., & Siegel, A. I. The rationale and application of job suitability 
as a basis for the evaluation of training. Personal Psychology, 1962, IS (3), 

Demonstrates a technique developed by Applied Psychological Services 
which employed suitability for the job” as a basis for tr ain ing evaluation 
by: ( 1 ) describing a method for quantitatively summarizing suitability, 
and (2) illustrating the application of the method to data collected on 
tcchmcians in four Naval jobs. 

Schutz, R. E. Methodological issues in curriculum research. Review of Edu- 
cation Research,. 1969, 39 (3), 354-366. 

Describes good methodology as one that can distinguish between myth * 
ology and reality, strong inference and weak inference, science and 
technology, and between comprehensive knowledge about education 
and education as a conjunction domain. A search of the literature shows 
that previous ‘methodology’ was only typology-building which, in the 
futive, will be supplanted by a form which has been produced across 
d^ciptoes in the social sciences for use by people at the journeyman level 
of training m curriculum specialization. 
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Siegel, A. I., & Schultz, D. G. Evaluating the effects of training. Journal of the 
American Society of Training Directors, September 1961, 52-54. 

Advocates th": legitimacy of using job performance as a cr terion for meas- 
uring training effects. The empirical emphasis in evaluation is defended. 

Skager, R. W., & Hroadbent, L. A. Cognitive structures and educational evalu- 
ation. Center for the Study of Evaluation, Report No. 46, University of 
California at Los Angeles. 

Examines the possibilities of cognif'vc measures as criteria in the evalu- 
ation of instructional programs. Represents a search of the literature for 
descriptions of the task and for findings of relationships between cogni- 
tive measures and other influences. 

Smith, R. C>., Jr. An annotated bibliography on proficiency Tneasttretnent for 
training^ quality control. Alexandria, Virginia: The George Washington 
University, Human Resources Research Office, 1964. 

An atmotated bibliography for use as a foundation for a quality control- 
iw-training manual. One hundred and one references are listed alpha- 
betically under five categories. 

Soik, N, An evaluation of a human relations training program. Journal of the 
American Society of Training Directors, Ma^ch 1958, 34-49. 

Attempts to evaluate the human relations program given at the Alien- 
Brad Icy Company. Four considerations were investigated: (1) the rela- 
tionship of individual characteristics to hirnian relations knowledge, (2) 
the contribution of one program to increased knowledge of the trainee, 

(3) the relationship of class participation to increased knowledge, and 

(4) the need for additional training. Training consisted of guided discus- 
sion, lecture, role playing, case studies, buzz groups, and movies. Three 
measuring instruments were used to evaluate the effects of training: the 
“Supervisory Inventory on Human Relations” (S.I.H.R.), comment 
sheets, and **The Human Relations Questionnaire.** 

Stake, R. E. The gcncralizability of program evaluation: The cotimtenance of 
education evaluation. Teachers College Record, 1967, 68 (7), 523-540. 

An original and provocative treatment of the problem of formal evalu- 
ation which provides the foundation for the development of a plan for 
evaluating educational inputs rather than educational outputs. The author 
contends that description and judgment are necessary for the u.: Jerstand- 
ing or educational programs, and that there is a need for data banks for 
the^ documentation of intelligence about antecedent conditions, trans- 
actions, and intents as well as ‘goals* and ‘objectives.* 

Stake, R. E. Testing in the evaluation of curriculum development. Rexnew of 
Educational Research, 1968, 38 (1), 77-84. 

Discusses curriculum evaluation development as distinguished from the 
measurement of individual aptitudes. 

Stake, R. E Gencralizability of program evaluation: The need for limits. 
Educational Products Report, 1969, 2 (5), 39-41. 

Discusses program evaluation which offers principles for the wridng of 
an evaluation plan in terms of whom it serves, what to ask, what will be 
measured, and what the range of findings may cover. 
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Scephenson, H. Evaluating human relations training. Personnel Administra^ 
July- August 1966, 34-39. 

Compares two programs for their applicability to training in human 
relations. Type 1 programs use a combination of the following tech- 
niques: lecture-discussion, role playing, seminar, movies, business and 
computer games, and outside study. Type 1 engages the trainees for a 
few hours a week either at the office or a centrally designated place. A 
Type 2 program normally involves a concentrated one- to two-week 
session away from the work cnvirorunenc. 

Stufficbeam, D. L. Evaluation as enlightenment for decision-making. Paper 
presented at the Working Conference on Assessment Theory sponsored by 
the Commi^ion on Assessment of Educational Outcomes, the Association 
for Supervision and Curriculum Development. Sarasota, January 19, 1968. 
Discusses the definition of the contemporary status of art in educational 
evaluation. Other possibilities for educational evaluation inclu^-e a general 
definition of evaluation, an outline of four strategies for educational 
change activities, and strategics for the development of evaluation design 
structure. 

Tarnopol, L. Evaluate your training program. Training and Development 
Journal, March-April 1957, 17-23. 

Presents a brief justification of ti'aining evaluation. 

Thisdell, R. A. Why not measure training results? Journal of the American 
Society of Training Directors, October 1959,9-12. 

Rationale for mc^uring training results. Two listings are given: one for a 
variety of ways in which training needs can be assessed and another for 
the ways^ in which training can be evaluated. The reasons training meas- 
urement is avoided arc presented along with counter arguments. 

Tracey, W. R. Evaluating training and development systems. American Man- 
agement Association, Inc., 1968. 

Presents a comprehensive overview of the principles and criteria for 
evaluation which includes philosophy and goals, management, plant and 
facilities, curriculum, and instructional support. A final portion of the 
book is concerned with a guide to rating standards. 

Training methodology — Part II: Planning and administration — an annciaied 
bihliogyaphy. Washington, D, C.; U. S. Government Printing Office, Public 
Health Service Publication No. 1862, 1969, 42-65. 

Ninety-six annotated references on the evaluation of training aie provided 
and categorized according to the following areas: evaluation — tests and 
meastirement (general), evaluation — process, evaluation — specific pro- 
grams, evaluation — specific methods, and evaluation — specific devices. 

Trow, A4. Methodical problems in the evaluation of innovation. Syriiposium 
presented at the Center for the Study of Evaluation, Report No. 31, May 
1969. ^ 

Discusses innovations hi the curriculum and modes of teaching and leam- 
ing. Emphasis is given to the characteristics being studied and assessed, 
and to the social context from which they emerge. 

Tyler, R. W. The functions of measurement in improvmg instruction. In E. F. 
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Lindquist (Ed.) Educatiovuzl measurement. Washington, D. C.: American 
Council on Education, 1951, 47-67. 

Discusses evaluation at it assists in selecting objectives, content, learning 
experiences, and procedures of instruction, supervision, and administra- 
tion. Conditions are described that are favorable to evaluation in helping 
instruction. 



Underwood, . J. Evaluation of laboratory-method training. Journal of the 
Amerrcan Society of Traming L>irec£o?Sj May 1965, 34-41. 

Evaluates through experimental prcccdures the effectiveness of sensitivity 
training by the T-Group method. Control and experimental groups were 
compared for interpersonal, personal, and nonpersonal i*u*ictioning. 

Vogels, D. S. Jr. An evaluatioii of a management training course. Journal of 
the Atnerican Society of Training DirectorSy January 1958, 44-51. 

Evaluates ’.nanagement rxaining course conducted at Griffith Air Force 
Sase, jNcw York. Pleasures of the effectiveness of training were accom- 
plished with the use of questionnaires. 

Wakely, J. H., & Shaw, M. E. Management training — an integrated approach. 
Training Directors Joumaly 1965, 19 (7), 2-13. 

Describes simulation in the training program of a new plan and evaluates 
the use of the laboratory method. Training in the program should be 
realistic; the sta^ should cooperate interdependently. There is a brief 
outline of all the components of the program. Four surveys were taken 
and four evaluations were made on four different di'*.*^nsions. The results 
are recounted. 

Walbesser, H. H. Curriculum by means of behavorial objectives. Journal of 
Kesearch in Science Teachingy 1963, 7, 296 501 . 

Reports the design and rationale for the evaluation study of the new cur- 
riculum j^terials for elementary school science being developed by the 
Comnussion on Science Elducation of AAAS. The evaluation process 
consisted of behavioral specifications of the curriculum objectives, their 
measurement by means of a checklist of competencies, and measurement 
of behavior by which scientific processes arc characterized. Since the 
aim of writing experimental material is the shaping of certain student 
bcliaviors, measures related to that behavior were included. 



Weiss, C. H. Evaluation of staff training. Welfare in Reineavy 1965, 3. (3), 

11-17. » » V » 



Discusses the purposes of the program of training evaluation, how results 
are_ used, and their effect upon the evaluation method. The evaluator- 
trainer relationship is also considered. 



Wherry, R. J. The past and future of criterion evaluation. IPersortnel Psy- 
chology, 1957, 10, 1-16. 

• - _ 

Di^usses R. M.. Bellows* list of criteria for evaluation which includes: 
reliability, accessibility and cost* acceptability to the sponsor. In a latter 
portion of the article two other criteria arc discussed. 



Wiley, D. E. The design and analysis of evaluation studies: Comments and 
suggestion. Symposium presented at the Center for the Study of Evaluation, 
Report No. 28, May 1969. 
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definition of evaluation. Evaluation assessment and 
appraisal are made di^nct from one another. The separate elements of 
evaluation are made cle^ alongr with their relationship to design, an..lysis 
and measurement of evaluation. ^ 

Wilson, C. L. On^he-}ob and operational criteria. In R, Glaser (Ed.) Training 
research and education. Pittsburgh: University of Pittsburgh piess, 196 :^ 

Trainer must have feedback about on-the-job performance before there 
Vr training. Learning theory offers a model for the 

requirements f^ three performance measurements which vary in degree 

specific behavior elements, gross per- 
formance, nnd ^Iperformance are categorized as operational pcrfo^- 
ance measures. Tests constructed to resemble the regular job offer useful 
where there is a work sample whose reliability has been obtained. 
Raungs are not of great utility m training studies because they are not 

daSfor" observers do not have sufficient 

Development 

An attempt is made by the author to examine critical features of the evalu- 
A programs that relate conceptual uuiterial to participants. 

for evaluating conceptual training programs is 
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